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ABSTRACT 
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design questions from theory, research, evaluation, and practice; (5) 
involve critical reflection on program philosophy and goals; (6) give 
prominence to the processes of teaching and learning; (7) capture a 
range of learner and program outcomes; (8) require a variety of 
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functions; and (10) be systematic and systemic. (131 references) 
(SK) 



Information Series No. 338 



ADULT LITERACY EDUCATION; 
PROGRAM EVALUATION AND 
LEARNER ASSESSMENT 



Susan L. Lytic 
Literacy Research Center 
University of Pennsylvania 



Marcie Wolfe 
Institute for Literacy Studies 
Lehman College, City University of New York 



[/this documftnl hns hOHn rffpfOduLtnl as 
^ fijceived (fom the person m orqnni^ntion 

Ofigtnaliny t| 
C Minof changes havi^ been made to improve 
reproduction quality 

■ f'oinlsntvieworopiniuMssl^iteM' ill istiocu 
menl do not nec*'sr..irily represenl ofliCjai 
OPHI position 01 potiny 



ERIC Clearinghouse on Adult, Career, and Vocational Education 
Center on Education and Training for Employment 
The Ohio State University 

1900 Kenny Road 
Columbus, Ohio 43210-1090 



U.S. DEPARTMENT OF EDUCATION 

()Mif:e ol tducattonal Mewiirch iuid Iniprovemen! 




EDUQfATlONAl. Hf SOURCE S INFORMATION 
/ Cf-NTf-R (tniQ 



1989 



BEST COPY AVAILAbLL 



FUNDING INFORMATION 



Project Title; 



ERIC Clearinghouse on Adult, Career, and Vocational 
Education 



Contract Nuraben 



RI88062005 



Act under Which 
Administered: 

Source of Contract: 



Contractor: 



Executive Director: 



Disclaimer: 



Discrimination 
Prohibited: 



41 use 252 (15) and P.L 92-318 

Office of Educational Research and Improvement 
U.S. Department of Education 
Washington, DC 20208 

The Center on Education and Training for Employment 
The Ohio State University 
Columbus, Ohio 43210-1090 

Ray D. Ryan 

This publication was prepared pursuant to a contract with the 
Office of Educational Research and Improvement, U.S. Depart- 
ment of Education. Con'jactors undertaking such projects 
u' .der government sponsorship are encouraged to express freely 
their judgment in professional and technical matters. Points of 
view or opinions do not, Aerefore, necessarily represent official 
U.S. Department of Efiucation position or policy. 

Title VI of the Qvil Rights Act of 1964 states: "T^o person in 
the United States shall, on the grounds of race, color, or nation- 
al origin, be excluded from participation in, be denied the 
benefits of, or be subjected to discrimination under any program 
or activity receiving federal financial assistance." Title DC of 
the Education Amcndmc its of 1971 states: "No person in the 
United States shall, on the basis of sex, be excluded from par- 
ticipation in, be denied the benefits of, or be subjected to dis- 
crimmation under any education program or activity receiving 
federal financial ii^sistancc." The ERIC Qcaringhouse project, 
like every prograiia or activity receiving financial assistance from 
the U.S. Department of Education, must be operated in compli- 
ance with these l aws. 



11 



ERIC 



TABLE OF CONTENTS 

LIST OF TABLES AND FIGURES v 

FOREWORD vii 

EXEOJnVE SUMMARY ix 

INTRODUCTION 1 

CONTR/VSTING PERSPECTIVES ON ADULT LITERACY 

JiDUCATION 5 

ItitiOductioD S 

Adults as Learners 5 

Concepts of Literacy 7 

Educational Contexts 12 

Conclusion 16 

RESOURCES FOR PLANNING PROGRAM EVALUATIONS 19 

Introduction 19 

Program Surveys 20 

• Handbooks 22 

Instruments 24 

Policy Studies 24 

Literacy Program Evaluatioms 28 

Broad-based Systemic Approaches 35 

Conclusion 37 

LEARNER ASSESSMENT 41 

Introduction , 41 

Standardized Testing 42 

Materials-based Assessment 47 

Competency-based Assessment 48 

Participatory Assessment , 51 

Conclusion 57 



• • • 

m 



ERIC 



/I 



IMPLICATIONS AND FUTURE DIRECTIONS 



Evaluation, Assessment, and Educational Policy 

A Framework for Program Evaluation and Learner Assessment 

in Ailrlt Literacy Education 

Conclusion 

REFERENCES 



iv 



UST OF TABLES AND FIGURES 



Table 

1. APPROACHES TO PROGRAM EVALUATION 

2. APPROACHES TO LEARNER ASSESSMENT 

3 CRITICAL FEATURES OF ADULT LITERACY PROGRAM 
EVALUATION 

figure 

L Contrasting perspectives on adult literacy education: 

some questions for programs 



FOREWORD 



The Educational Resources Infonnation Center Gearinghouse on Adult, Career, and Voca- 
tional Education (ERIC/ACVE) is 1 of 16 clearinghouses in a national information system 
that is funded by the Office of Educational Research rmd Improvement (OERI), U.S. 
Department of Education. This paper was developed to fulfill one of the functions of the 
clearinghouse-interpreting the literature in the EKIC database. This paper should be of 
interest to adult education policy makers, administrators, and researchers, as well as adult 
literacy program personnel. 

ERTC/ACVE would like to thank Susan L Lytle and Marcie Wolfe for their work in the 
preparation of this paper. Dr. Lytle is Associate Director of the Lit eracy Research Center 
at tne University of Pennsylvama, where she also serves as direaor of the Philadelphia 
Writing Project and the Program in Reading/Writing/Uteracy. She is directing the ongoing 
Adult literacy Evaluation Project (ALEP) funded by the Philadelphia National Bank. 
ALEP is a collaborative project linldng participatoiy assessment and research on adult 
literacy development. 

Marcie Wolfe is Director of Adult Programs at the Institute for Literacy Studies, Lehman 
College, City University of New York. She is directing the Adult Educators Development 
Project, which seeks to develop program- and classroom-based assessment for adult literacy 
learners. She has also directed st^ development institutes in the teaching of writing for 
New York City teachers in schools and nontraditional settings. 

The authors would like to thank Paul R. Allison, Jolley Christman, Vivian Gadsden, John 
Garvey, Karen Griswold, Richard L. Larson, Nancy Leslie, James H. Lytle, Dianne 
Marenco, Katherine Schultz, and Richard Sterlmg for their responses and contributions to 
th'* monograph. In addition, they are grateful to those literacy educators and researchers 
who provided important references and materials not widely disseminated. ' 

The following people are also acknowledged for their critical review of the manuscript prior 
to publication: Thomas Valentine, Associate Professor, University of Georgia; Hanna 
Arlene Fing^ret, Director, North Carolina Center for Literacy Development; Judith A 
Alamprese, Director of Education and Training, COSMOS Corporation; Susan Klein, 
Cleannghouse Monitor, OERI; and James M. Weber, Senior Research Specialist, the 
Center on Education and Training for Employment 

Publication development was coordinated by Wesley Budke and Susan Imel. Sandra Kerka 
edited the manuscript, and Janet Ray served as word processor operator. 

Ray D. Ryan 
Executive Director 
Center on Education and 
Training for Employment 



vii 



ECUnVE SUMMARY 



Increased attention to adult literacy programs in recent years has not been accompanied by 
resources to support systematic evjQuation of these efforts. The constituents of literacy 
programs-learners, administrators, staff^ and sponsors-need reliable informatica about 
program quality and effectiveness in order to understand and improve the critical 
dimensions of service provision. Evaluation data are essential for accountability, 
improvement of practice, and expansion of the kno wledge base on adult literacy education. 

Evaluation and assessment reflect fundamental beliefs about learners, literacy, and 
educational settings. The design of evaluation can be shaped by addressing certain 
questions about these three areas, including the following. 



Adults as Learners 

o How is the program designed specifically for adult learners? 

o How congruent are teachers' and leamerF' concepts of success? 

o What roles do adult learners and staff play in setting and revising program goals? 



Concepts of Literacy 

o What is the program's working definition of literacy (literacy as skills, tasks, practices, 
critical reflection)? 

o What information about the cultures, communities, and expectations of adult learners 
has been used to reach this definition? 



Educational Contexts 

o To what extent does the program's design relate to adult learning in everyday life? 
o Is the program individually or community oriented? 

o How does the program provide opportunities for teachers to expand their instnictional 
repertoires? 



ix 



Resources for planning program evaluations include program surveys, handbooks, 
instruments, and policy studies. Representative evaluatioja*: of volunteer programs, 
statewide adult basic education, community-based programs, technology-oriented programs, 
and workplace approaches illustrate some of the issues involved: (1) program goals jmd 
mission are subject to scrutiny and change, (2) data on the processes of teaching and 
learmng are essential, (3) expanded outcome measures for learner progress are needed, and 
(4) the roles of staff; managers, learners, and external evaluators all have an impact on the 
processes and outcomes of evaluation. 

Four major types of approaches to learner assessment have been identified: 

0 Standardized testing is norm referenced, considers literacy to be reading skills, is cost 
effective and independent of curriculuuL 

0 Materials-based assessment is commercially available, is related to progress in 
predetermined materials, and follows a systems management model. 

0 Competency-based assessment involves specific real-life tasks, predetermined 
performance standards, a continuum of difficulty, and a broad range of strategies. 

0 Participatory assessment views literacy as practices and critical reflection, gives learners 
an active role as co-investigators, and involves a range of texts, tasks, contexts, and 
strategies. 

A framework for program evaluation and learner assessment in adult literacy education has 
10 critical features: 

1. Program evaluation in adult literacy e ducation should be conducted both externally 
and internally. 

2. Program evaluation should be both formative and summative. 

3. Program evaluation and learner assessment should involve learners and staff in a 
participatory process. 

4. Questions for the design of program evaluation should be generated from theoiy, 
research, and evaluation as well as from practice. 

5. Program evaluation should involve critit al reflection on pf ogram philosophy and goals. 

6. Program evaluation should give prominence to the processes of teaching and learning. 

7. Evaluations should be designed to captui e a range of learner and program outcomes. 

8. Program evaluation and learner assessment require a variety of methods for collecting 
data over time. 



V./ I ; f) 
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9.. Evaluation and assessment should be integrated with program functiom. 

10. Program evaluation should be systematic and systemic, enabling stal<:eholders to make 
comparisons within and acioss programs and contexts. 

Information on evaluation and asseRsment of adult literacy nmgrams msy be found in the 
ERIC system using the following descriptors: Adult Basic Education, Adult Educators, 
Adult Learning, 'Adult Literacy, Competency Based Education, Educational Policy, 
•Evaluation Methods, 'Evaluation Research, Fimctional Literacy, 'Literacy Education, 
Measures (Individuals), 'Program Evaluation, Reading Tests, Standardized Tests, 'Student 
Evaluation, Student Participation. Asterisks indicate descriptors having particular relevance. 



10 



INTROJUCnON 



Over the past decade, increased attention 
to the demand for adult literacy services 
has resulted in more funding for pro- 
grams, in greater numbers of people par- 
ticdpati^ig, and in the rapid development 
of new sites for tutoring and classes, 
libraries, conununity organizations, 
churches, unions, businesses, and corpora- 
tions, to same just a few, have joined with 
school s>'stems, coP^ges, and universities 
in attempting to ao .ess the diverse needs 
of learners in a variety of settings. Taken 
together, these efforts signal a strong com- 
mitment by different groups and individ- 
uals to create effective educational pro- 
grams for adult learners. In this period of 
growth, however, there have been limited 
resources~and few incentives~to evaluate 
these efforts in a systematic way. 

For the most part, adult literacy program 
evaluations have been conducted to re- 
spond to the needs of funders or sponsor- 
ing organizations for data relating pro- 
gram goals and outcomes (Lerche 1985a). 
Such data have typicaUy included demo- 
graphics, numbers of instructional and 
contact hours delivered, allocation of 
. resources, and gains on standardized tests. 
State agencies funding programs through 
the Adult Education Act of 1988, for 
example, now must evaluate at least one- 
third of grant reciments on the basis of 
such data. Most p i igrams sec themselves 
as having few ira'.ntives, limited funds, 
and insufficient resources to go beyond 
these requirements. Only a few programs 
have had access to technical assistance to 



design and implement more sophisticated 
evaluation processes. Some programs 
have been developing their own 
approaches to learner assessment but in 
most cases, have still been required to 
administer tests and report progress based 
on a single score. Many feel that this 
approach does not capture much of what 
is learned in programs and is inadequate 
for providing information about program 
strengths or weaknesses. The problem is 
not just the dependence on a single indi- 
cator; in most cases, measurement of 
learner outcomes has substituted for more 
comprehensive and rigorous evaluation of 
whole programs. In addition, much of 
what has been learned by programs about 
successful practice has occurred informally 
and been used internally, so that knowl- 
edge gained has often not accrued across 
sites and geographical boundaries. 

As a consequence, most would agree that 
the remarkable growth in services has not 
been accompanied by a parallel growth in 
stakeholders' knowledge or understanding 
of many critical dimensions of service pro- 
vision and the patterns of literacy develop- 
ment by adults in different contexts. 
Stakeholders are those audiences for eval- 
uation who have some stake or interest in 
the performance or outcomes of a pro- 
gram (Cuba and Uncohi 1981). For adult 
literaQT programs, stakeholders include 
those most directly involved in the pro- 
gram, for example, administrators, staff, 
and adult learners, as weU as umbrella 
organizations-larger networks to which 
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literaiy prcgrams nay be closely or 
loosely aLlgned (for example, Association 
for Community Based Education, literacy 
Volunteers of America, Laubach LiteraQr 
Action, or a citywide organization), 
and/or sponsoring or funding organiza- 
tions such as foundations, corporations, 
and government agencies. All of these 
constituencies seek information about 
program quality and about the ways in 
which existing services can be Linproved, 
and thus all would benefit from more 
complex and sensitive program evaluation. 

Data from the evaluation of literacy pro- 
grams, including assessment of learners, 
are essential for several purposes. The 
first are clearly accountability: demon- 
strating to stakeholders \'hat programs 
have successfully used theii' resources to 
achieve identified outcomes. Issues of 
accountability have become both roc.e 
complicated and more critical, in part 
because there is a lack of agreement in 
the field about common criteria for mea- 
suring success, either of programs or 
individual learners. The second purpose 
relates to the improvement of practice: 
evaluation and assessment are processes 
of inquiry that have the potential to iden- 
tify and help address essential qu'^^stions 
about teaching and learning, program 
management, and the allocation of 
resources. There are few structures out- 
side programs offering professional devel- 
opment for adult literacy educators in 
these areas. Collaborative planning and 
implementation of strategies for evalua- 
tion and assessment can provide a context 
for this development within programs and 
strengthen networking across programs as 
well. Finally, the thoughtful analysis and 
interpretation of data from program eval- 
uation and learner assessment can contri- 
bute to the knowledge base on adult lit- 
eracy education. A variety of perspectives 
are needed to advance imderstanding of 



critical dimensions of literacy education. 
Some important information can come 
from collaboration among educators and 
learners within and across programs 
focused on issues in evaluation and assess- 
ment Other perspectives can come from 
collaborations between practitioners and 
university-based researchers and consul- 
tants. Such collaborations can reduce the 
Isolation of adult literacy programs fiom 
each other as well as increase their access 
to theoiy and research in the field. 

This monograph is intended primarily to 
make accessible to adult literacy educators 
the current literature in the field related 
to program evaluation and learner assess- 
ment. In addition, this information may 
be of value to policy makers, funders, and 
researchers who are committed to improv- 
ing the quality of literacy education for 
adults. The second chapter reviews con- 
trasting perspectives on adults as learners, 
concepts of literacy, and contexts for 
education to suggest some of the ways in 
which dedsions about evaluation and 
assessment may be informed by underlying 
beliefis or assumptions. In chapter 3, the 
current literature related to program 
evaluation is critically reviewed The 
intent is to explore the usefulness of this 
literature for the design and implementa- 
tion of different strategies as well as to 
situate learner assessment withm a 
broader view of evaluation. Chapiter 4 
contrasts approaches to learner assessment 
currently available to suggest their 
distinctive contributions to the under- 
standing of what can and shoiUd be 
assessed and the relative appropriateness 
of different methods for accomplishing 
different purposes. The final chapter 
presents a framework for relating program 
evaluation and learner assessment, for 
critiquing current efforts, and for planning 
new directions that address i£sues of 
accountability, the improvement of prac- 



tice, and the need for a richei base of 
knowledge in the field. 
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CONTRASTING PERSPECTIVES ON ADULT 
LITERACT EDUCATION 



Introduction 

Evaluation and assessment involve formu- 
lating questions and seeking evidence in 
order to make judgments about the qual- 
ity of programs and the accomplishments 
of learners. These questions, and the 
evidence gathered to study them, in turn 
depend on fundamental assumptions or 
beliefs about adults as learners; about the 
acquisition, development, and uses of lit- 
eracy; and about how different educa- 
tional contexts promote learning. These 
assumptions or beliefs have been the sub- 
ject of ongoing debates and discussions 
among literacy educators, policy makers, 
and researchers interested in program 
design and practice, and they need to be 
prominent in debates and discussions 
about the design of program evaluation 
and learner assessment. 

To highlight the ways in which the pro- 
cesses and outcomes of evaluation and 
assessment reflect particular beliefs about 
learners, literacy, and educational settings, 
this chapter explores some contrasting 
perspectives from research and practice. 
Although a comprc hensive review of per- 
spectives on these three areas is not 
within the scope of this monograph, this 
review is intended to invite discussion 
about the purposes of adult literacy 
education and its social and political 
meanings and consequences. These dis- 
cussions are central to decisions about 



evaluation and assessment, as they chal- 
lenge some assumptions and stimulate 
considerations of alternatives. The topics 
and issues included in this review are also 
intended to stimulate questions useful in 
the analysis and critique of the litr;rature 
in chapters three and four. 



Adults as Learners 

Debates about the adults thought to be in 
need of literacy education center on their 
characteristics, intentions and goals for 
seeking instruction, and modes of learn- 
ing. In contrast to popular perceptions, 
researchers argue that the characteristics 
of adults in need of literacy education do ' 
not necessarily fit the images of depen- 
dency, weakness, and failure depicted in 
media stereotypes (Fingeret 1983; Smith- 
Burke, Parker, and Deegan 1987). Rath- 
er, they operate within complex social 
networks in which they are interdepen- 
dent, offering skills of their own in 
exchange for the literacy skills of others. 
Adults lacking literacy skills are therefore 
as varied in interests, abilities, and self- 
perceptions as any other group within the 
population. Adults are often further char- 
acterized as "suffering" from illiteracy, 
which in turn contributes to poverty and 
an array of other social hardships. Al- 
though it is true that large numbers of 
undereducated adults encounter many per- 
sonal and economic problems, there is 
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little evidence that literacy alone either 
causes this situation or remedies it (Graff 
1987; Hunter and Hamian 1979). 

Pervasive beliefs about the needs of adult 
literacy learners affect how literacy pro- 
grams are conceptualized and how adults' 
motivations for participation are perceived 
(Johnston 1985). Not only do teachers' 
expectations of learners vary, but teachers 
and learners sometimes emphasize differ- 
ent areas of achievement. In contrast to 
the functional and cognitive achievements 
emphasized by teadbers in the United 
Kingdom, for example, Chamley and 
Jones (1979) found that learners attended 
literacy programs in order to attain per- 
sonal and social goals, such as developing 
better family and work relationships and 
participating in civic duties. In a study of 
North Carolina adult basic education 
(ABE) programs (Fingeret 1985), teachers 
viewed success mainly in terms of move- 
ment through the schooling hierarchy, 
whereas learners were found to consider 
success as their increased ability to apply 
reading, writing, and math to the demands 
of their daily lives and as positive changes 
in their relationships with family and 
friends. Their ability to change the cir- 
cumstances of their lives, or to attempt 
such a change, was how they measured 
success. At the same time, learners also 
expressed "schooled" articulations of suc- 
cess in terms of gains in subsldlls and 
social mobility. Because data about the 
sources of adult motivation to achieve in 
school-like ways is elusive, studies that 
confkm adults' intention to seek their 
General Educational Development (GED) 
certificate leave some important questions 
unanswered. 

These debates about intentions and goals 
are related to the current interest in work 
force literacy. It has become a comm'. n- 
place that entry-level positions are 



increasingly more complex and therefore 
require better literacy and computational 
skills. Adults entering job training pro- 
grams may have expectations about job 
mobility and opportunities for substantial 
changes in their life circumstances that 
may not be satisfied. Rather than 
empowering more of the disenfranchised, 
filling entry-level slots may serve to 
maintain the current distribution of power 
(Fingeret 1990). Furthermore, although 
many adult learners do seek literacy 
education as a route to employment or 
advancement in the workplace, other 
participants in workplace programs bring 
or develop a wider range of interests and 
goals (Darkenw£dd and Valentine 1984; 
Hikes 1988b). This situation raises 
questions about the breadth of curriculum 
in work force-related programs and the 
ways in which they can respond to partici- 
pants* evolving concerns. 

The debates about adult characteristics 
and motivations are related to debates 
about adult modes of learning. Much of 
ihis discussion stems from Knowles' (1979) 
theory of andragogy, which posits a set of 
assumptions about adult learning related 
to self-direction, experience and prior 
knowledge, immediacy and applicability of 
learning to roles in society, and a focus on 
problems rather than on subject matter. 
Andragogy is defined by Knowles as '*the 
art and science of helping adults learn" 
(p. 8). Knowles can be credited with 
popularizing a view of adult development 
that distinguishes adults from children and 
with focusing attention on the learner in 
adult education programs. 

The critique of andragogy focused initially 
on whether it is possible to make such 
sharp distinctions between how adults and 
children leant Later, other critics raised 
questions about Vbether andragogy is a 
learning theory, a philosophical position. 
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a political reality, or a set of hypotheses 
subject to scientific verification'' (Cross 
1981, p. 225). Brookfield (1986), for 
example, questions the assumption that 
adult learning is always self-directed, 
arguing instead that self-directedness is 
cultur^ and class-specific. One problem 
with andragog/ as a theoiy of adult learn- 
ing, some argue, is that it obscures the 
considerable diversity of adults in culture, 
social class, and educational opportunity. 

The literature on adults as learners, 
briefly described, raises provocative 
questions for adult literacy programs. 
Tbese include such issues as whether adult 
programs replicate relationships between 
teachers and learners associated with 
traditional schooling, whether programs 
attempt merely to respond to learners' 
expressed goals or participate in shaping 
them, and whether programs recognize the 
complex motivations Uiat learners often 
bring to their decision to enroll. Further, 
the available literature on adults as 
learners does not distinguish between 
adult literacy and other categories of adult 
education. Research that focuses specific- 
ally on those adults attending literacy 
programs could contribute much to discus- 
sions of program philosophy, staff devel- 
opment, and strategies for involving adults 
more actively in all aspects of the pro- 
gram. Additional research on the spon- 
taneous acquisition of literacy in family 
and community contexts (Reder 1987; 
Reder and Green 1985) provides other 
important data about literacy learning by 
and among adults. 



Concepts of Literacy 

There is little agreement in adult literacy 
education or in broader educational 
circles about what is meant by literacy. 
The task of defining it has become even 



more complex as the use of this term in a 
wide variety of contexts becomes more 
common (see, for example, Venezky, 
Wagner, and Ciliberti 1990). In this sec- 
tion many of the finer distinctions among 
these definitions are collapsed into four 
broad categories in order to highlight 
some of the important debates and ques- 
tions related to literacy acquisition, devel- 
opment, and use. 



literacy as Skills 

In the first conception, literacy is defined 
in relation to the traditions of reading 
instruction in K-12 schools and thus to the 
attainment of grade levels as indicated by 
performance on standardized tests. From 
this perspective, literacy is generally 
equated with decontextualized skills 
regarded as stable across texts and con- 
texts. Whether or not grade levels are 
explicitly designated, educators working 
within a skills framework describe pro- 
gress hieraithically and refer to bamers 
as being at particular levels. 

This model of literacy can be traced to 
developments in the late 19th and early 
20th centuries related to group testing and 
to the call in the educational community 
at that time for scientific and objective 
measures of individual reading achieve- 
ment. The resulting paradigm (for exam- 
ple, tests of silent reading and systematic 
analyses of oral reading errors) has per- 
sisted into present literacy education for 
adults as well as children. Chall (1990), 
for example, describes literacy as an array 
of skills that constitute levels of literacy 
achievement applicable to adult Hteracy 
programs. She considers "functional liter- 
acy" as corresponding to the fourth- to 
ei^th-grade reading levels and as charac- 
terized by the ability to read the local 
paper or easier articles in magazines. 



Although there is widespread critique of 
decoDtextualized skills and of reading 
levels applied to adult learners (Johnston 
1985; Stedman and Kaestle 1987; Sticht 
1990), the pervasiveness of the school 
model in the prior experience of most 
teachers, tutors, and learners makes its 
presence as a framework for instruction in 
adult literacy programs unsurprising. 
Even when the use of standai dized tests is 
minimized, curriculum often implic'tly 
reflects the belief that effective literacy 
instruction involves breakmg reading and 
writing down into their component parts 
and teaching them sequentially. Much 
time and attention are often given to the 
processes of decoding and encoding. 

The equation of literacy with reading 
skills is compatible widi what Street 
(1984) calls the "autonomous model," the 
idea tihat literacy is a neutral and objec- 
: - 0 skill or set of skills independent of 
any specific social context or ideology. 
Cook-Gumperz (1986) refers to this view 
of literacy as smgular and stratified, 
designed to be tau^t in schools as a set 
of universal, cognitive, and technical skills. 
Since adults who come to literacy pro- 
grams in the United States have most 
often attended school as children for some 
period of time, the underlying assumption 
that programs should diagnose specific 
deficits and prescribe instmctional 
remedies follows logically from people's 
prior ex ^,rience. The hegemony of this 
model Las many consequences, among 
them that literacy educators who subscribe 
to a whole language approach, for exam- 
ple, are faced with an inherent contradic- 
tion between their own view of literacy 
learning and learners' more traditional 
expectations. In such a situation, meeting 
learners' expectations may conflict with 
exercising professional judgment 



Literacy as Tasks 

The view of literacy as tasks is associated 
with functional literacy, which in turn has 
a variety of interpretations. Definitions of 
functional literacy reflect shifting stan- 
dards and emphases. One definition of 
this term links amounts of schooling with 
reading and writing attainments (Chall 
1990). Sometimes functional literacy is 
understood as an intermediate level of 
fluency-more complex than the ability to 
read street signs or write one's name, but 
less than "fiill Uteracy" (Levine 1982). 
Other definitions have evolved from 
surveys o: literacy attiuimient conducted 
over the last 20 years. The Harris surveys 
on adult functional literacy (1970-71, cited 
in Nafeiger et al. 1975) defined degrees of 
literacy as perff • ^f»iice on an array of 
representative daily tasks, that is, filling 
out forms related to specific real-worid 
transactions indexed for difficulty 
(Nafeiger et al. 1975; Venezky, Kaestle, 
and Sum 1987). 

Still other definitions of functional literacy 
have been more relativistic. Ideological in 
nature, these definitions situate functional 
literacy within the needs and characteris- 
tics of different groups and cultures. Gray 
(1956, cited in Levine 1982), in a survey 
of reading and writing for UNESCO, orig- 
inally defined it as "the acquisition of 
knowledge and skills in reading and writ- 
ing which enable one to engage in all 
those activities in which literacy is nor- 
mally assimiedTin his culture or group" 
(p. 249), although he later went on to 
identify 3 yeai: "1 schooling as the cri- 
terion needed to reach tLis level. Hunter 
and Harman (1979) view functional liter- 
acy sunilarly and fiuther stress the role of 
persons and groups themselves in setting 
criteria for what counts as functional in 
their lives. Finally, some researchers 
suggest that these many ways of defining 
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functional literacy point to a fundamental 
weakness in the concept, an "extreme elas- 
ticity of meaning" (Levice 1982) that 
allows for conflicting interpretations while 
seeming to represent widespread agree- 
ment Further consequences of the vari- 
ous meanings of functional literacy include 
the inability to identify its spediGc skills 
and the differences in estimates of func- 
tional illiteracy in the United States 
(Kirsch and Guthrie 1977-78). In Levine's 
own definition, functional literacy is 
related to specific kinds of information 
and particular kinds of information needs. 
This definition, which implies shifting 
distinctions between illiteracy and literacy 
depending on task and context, suggests 
that all people possess some degree of 
functional illiteracy. 

The Adult Performance Level (APL) Pro- 
ject ^1975) shifted the concept of func- 
tio lal literacy to the notion of functional 
c< <mpetency and emphasized "considerable 
background Imowledge** and "skills beyond 
document literacy" (Venezky, Kaestle, and 
Sum 1987). APL researchers used a 
national survey to develop a matrix of 
general knowledge areas and skills and 
the methods to assess these competencies. 
From this survey, they estimated the 
percentage of the population who wtre 
not functionally competent for "economic 
and educational success in today's society" 
(Northcutt 1977, p. 1). Althou^ suppon- 
ers state that the underlying strength of 
the APL concept is its focus on "rele- 
vance, practicality, applicability and 
immediacy," critics have pointed out that 
this notion of literacy as functional 
competency has at its center a funda- 
mental contradiction. It first proposes 
that competence is context-specific and 
then posits the same tasks for all persons 
(see, for example, James 1977). 
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Studies of functional literacy such as the 
APL and Harris surveys set the stage for 
more sophisticated concepts of literacy as 
tasks. Researchers such as Kirsch and 
Jungeblut (1986) have, in effect, combmed 
notions of skills and tasks. They empha- 
size "the multiple nature of literacy skills 
and [report] profiles of people at different 
skill levels on different tasks" (Venezky, 
Kaestle, and Sum 1987, p. 15). Defining 
literacy as "using printed and written 
information to iSinction in society, to 
achieve one's goals, and to develop one's 
knowledge and potential" (Kirsch and 
Jungeblut 1986, p. 1-8), the National 
Assessment of Educational Progress 
(NAEP) divided literacy into three areas 
meant to represent the categories of lit- 
eracy activities in which people are likely 
to engage at home, at school, and at work: 
prose literacy, document literacy, and 
quantitative literacy. The degree of diffi- 
culty of specific reading material was 
determined by the complexity of the task 
involved. Three variables-the number of 
features in the question that had to be 
matched to information in the document, 
the degree of conespondence between the 
wording in the question and that in the 
document, and the number of possible 
correct answers-were used to identify 
what it means to perform at particular 
levels on each of the scales. This notion 
of task is not limited to demonstrations of 
performance; it also seeks to capture the 
thinking required in the interaction among 
reader, task, and specific types of text. 

In contrast to the NAEF tasks, which were 
designed to be familiar and accessible to 
a wide range of people, other views of lit- 
eracy as tasks emphasize their context spe- 
cificity. In theu- research on literacy in 
the workplace, Sticht and Mikulecky 
(1984) identify and base instruction on 
tasks specific to workplaces and workers 
such as Comprehensive Employment and 



Training Act (C£TA) word-processing 
trainees, wastewater plant workers, and 
members of the Army. They demon- 
strated that success on work-related 
materials depends on the use of specific 
prior knowledge and particular strategies 
for technical reading and problem solving. 



Literacy as Practices 

The third perspective on literacy emanates 
from social and historical examinations of 
literacy in different cultures. To under- 
stand the social meanings of literacy, 
researchers working in an ethnographic or 
anthropological tradition have studied 
what language means to its users and how 
it is used by them. Studies of the func- 
tions and uses of literacy in different 
cultural contexts reveal distinctions among 
literacy practices of different cultures and 
communities (Heath 1983; Phillips 1972; 
Reder and Green 1985; Taylor and 
Dorsey-Gaines 1988). In addressing the 
social, political, and economic nature of 
literacy practices, this model of literacy is 
congruent with Street's (1984) "ideological 
model" in which practices differ from 
group to group within a society as well as 
from society to society. 

Heath's (1983) work in the Piedmont 
Carolinas can be seen as an exploration of 
the tensions between the assumptions 
about literacy from which middle-class 
teachers operate in schools and literacy 
practices as they appear in the homes and 
communities of the working-class students 
who attend these schools. Heath demon- 
strates that the different ways children 
learn to use langiiage arc dependent on 
the ways in which each < ommunity struc- 
tures its families, defines the roles that 
community members assume, and 
plays out the concepts of childhood that 
guide children's socialization within a 



particular cultm'al and religious milieu. 
Heath's work and more recently the 
ethnography of inner-city families by 
Taylor and Dorsey-Gaines (1988) provide 
a strong contrast to deficit models that 
emphasize the limited background some 
children (and adults) bring to learning. 
Instead, Heath and others (such as Reder 
and Green 1985) find that, although the 
uses of reading and writing are tied to the 
particular economic and cultural patterns 
of a community, families use literacy for a 
wide variety of purposes and audiencca 
and in diverse situations. 

literacy programs, however, often fail to 
acknowledge these various uses. In con- 
trast, Auerbach (1989) offers a "social- 
contextual" model of family literacy and 
asks, "How can we draw on parents' 
knowledge and experience to inform 
instmction?" (p. 1T7). Her suggestions, 
drawn from the University of Massachu- 
setts Family literacy Project, focus on 
increasing "the social significance of 
literacy in family life by incorporating 
community cultural forms and social issues 
into the content of literacy activities" 
(p. 177). These suggestions, rooted in 
Freirean problem posing (see next sec- 
tion) and whole language approaches, 
include co-investigations through reading 
and writing with adult learners into family 
literacy practices; community, workplace, 
and health care issues; and parental 
concerns. 

Gting evidence from both historical and 
sociological research, Mikulecl^ (1990) 
also points to the "overwhehning influence 
of contex^i on literacy purposes, demands 
and processes" (p. 26). The wide varia- 
tions reflect our pluralistic culture and the 
many different social contexts in v/nich 
literacy is used. Transferring literacy 
abilities from one context to another, 
Mikulecky argues, depends on similar 
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fonnats, social support networks, and 
knowledge of the background information 
needed to use literacy in another setting. 
Mikulecky cautions that differences in 
literacy use and function among different 
cultural groups have the potential to 
create situations in which the literacy of 
minority groups is devalued and becomes 
a justification for discrimination and 
gatekeeping. In responding to 
Mikuleck/s perspective, Fingeret (1990) 
warns that respect for literacy variation as 
socially situated maj' lead to an uncritical 
acceptance of a normi tive framework with 
no agenda for social <!hange. Analysis of 
the fonction and purpi^se of literacy activi- 
ties must be done at a societal level and 
action taken through social policy so that 
issues about literacy, power, and the redis- 
tribution of power can be addressed. 



Literacy as Critical Reflection 

Oosely linked to the perspective of liter- 
acy 35 practices, a fourth perspective 
regards literacy as a process of interpret 
ing the world and developing a conscious- 
ness of commonly held values, behaviors, 
and beliefs as socially and culturally 
constructed. The seminal work of Paulo 
Freire has contributed to the growing 
interest in this perspective in adult literacy 
education. Freire sees literacy as "reading 
the word" in order to ''read the world," 
that is, being able to use reading, writing, 
and discussion to reflect on the conditions 
of one's existence and ultunately to 
change them. Others refer to this per- 
spective as "critical literacy," defined as 
the ability of people to read in order to 
"to decode critically their personal and 
social world and thereby further their 
ability to challenge the myths and beliefs 
that structure their perceptions and 
experiences" (Giroux 1988, p. In this 
conception of literacy, issues of race. 



ethnicity, gender, social class, and history 
become both subjects and interpretive 
frames for reading and writing. 

In the adult education literature, this 
concept of literacy is most similar to 
Brookfield's (1985) notion of critical 
reflectivity. Adult education, he argues, 
should focus on helping adults take con- 
trol of their lives by identifying "the 
external sources and internalized assump- 
tions framing their conduct" and assessing 
them aitically. Brookfield makes a dis- 
tinction between education and trainbg. 
hi training, previously defined skills, 
knowledge, and behaviors are "trans- 
mitted" to the learner. Education, by 
contrast, requires individuals to consider 
alternatives and question conventional 
wisdoms. This occurs through dialogue 
among learners and between leamers and 
teachers. Brookfield emphasizes that an 
awareness of beliefs, values, and behaviors 
as culmrally constructed leads to the pos- 
sibility of individual change and collective 
action. 

The view of literacy as critical reflection 
contrasts with the notion of cultural lit- 
eracy, the proponents of which prescribe 
a body of Imowledge assumed to have uni- 
versal value and to be necessary for par- 
ticipation in this society (Bloom 1987; 
Hirsch 1987). Unlike critical Uteracy, 
which raises questions about what counts 
as knowledge and how it is generated, cul- 
tural literacy takes as given that particular 
texts and infocm'^tion are prerequisite for 
a shared understanding and preservation 
of societal values. In practice, adult edu- 
cators committed to critical hteracy struc- 
ture classes around the exploration of sig- 
nificant themes kJid texts in learners' lives. 
However, the concept of cultural literacy 
provokes debate about the relationships 
between the knowledge diverse learners 



11 



20 



bring and the choice of texts and content 
for literacy education. 

Program evaluation and learner assess- 
ment are groimded in particular assump- 
tions about the nature of literacy, its 
acquisition, development, and use. Al- 
though there is a consensus th&t the 
diversity of learners requires a similarly 
diverse array of options and programs, 
spedfic views of literacy have conse- 
quences for the range of opportunities 
available, or not avsdlable, to learners. 
An investigation into views of literacy held 
by program staff and learners may help to 
clsdfy goals, uncover discrepancies, and 
suggest alternatives. Such an investiga- 
tion, however, may also lead to conflicts 
that are not easily resolved. 



£du«;ational Contexts 
Learning out of School 

Although many adults in literacy programs 
bring wiih them histories of failure in 
school, they also bring with them diverse 
and often qui^e extensive experiences 
using print in tneir daily lives and some 
success as learners in domains outside of 
school. Researchers asking questions 
about how the nature of learning shifts 
depending on context have begun to look 
at ways in which learning occurs both in 
and outside formal educational settings 
(Resnick 1987). 

One important concept m this discussion 
is the distinction between acquisition and 
learning (Gee 1987; Krashen 1982). 
Acquisition is defined as the process of 
obtaining something without du-ectly 
focusing on learning it. This occurs 
through "ex'josure to models" within the 
course of dtily life and a "process of trial 
and error" (Gee 1987). Learning, on the 



other hand, involves "conscious knowledge 
gained through teaching" (Gee 1987). 
Learning is characterized by direct 
attention to explanation and analysis and 
to the attainment of some degree of meta- 
cognitive awareness about the item 
learned. Using these definitions, out-of- 
school literacy development might be said 
to be more closely aligned with acquisi- 
tion, as it occurs in the contexts in which 
people ordinarily fin^ themselves, and 
does not involve formal reaching. 

In describing informal, out-ot-school liter- 
acy development, Reder and Green (1985) 
distinguish be^veen two types. In the first 
variation, individuals teach others the 
skills and knowledge needed to manage 
iipecific literacy tasks. Although there 
have been some instances of the teaching 
of reading and writing by one member of 
the conmiunity to anoUier, the occasions 
for informal literacy teaching have more 
often occurred spontaneously in response 
to particular needs. In the second varia- 
tion, individuals receive help within their 
social networks, where friends and neigh- 
bors offer to complete literacy tasks in 
exchange for other skills (Fingeret 1983; 
Reder and Green 1985). Reder and 
Green stress that even in situations where 
there is little or no direct teaching and 
where the person needing help does rot 
learn to do a task as a result of the 
assistance, some learning about the spe- 
cific social uses of literacy does occur. 
These studies suggest that understanding 
adult literacy development in particular 
contexts is enhanced by an understanding 
of the "socially organized patterns of 
coUaboration" (Reder 1987) within a 
culture or community. 

Resnick (1987) makes distinctions 
between characteristics of learning in and 
out of school that have implications for 
adult literacy programs. She argues that 
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getting along in the world requires a 
"practical intelligence" that not only differs 
from school learning but that may matter 
more in daily life. Resnick identifies four 
distinctions betv een learning in school 
and out: 

Briefly, schooling focuses on the 
individual's performance, whereas 
out-of-school mental work is often 
socially shared. Schooling aims to 
foster unaided thought whereas 
mental work outside school usually 
involves cognitive tools. School 
cultivates symbolic thinking, whereas 
mental activity outside school 
engages directly with objects and 
situations. Finally, schooling aims to 
teach general skills and knowledge, 
whereas situation-specific competen- 
cies dominate outside, (p. 16) 

Resnick*s framework emphasizes the 
social nature of learning out of school, the 
use of tools, and its context specificity. 

Other research on everyday cognition 
(see, for example, Rogoff and Lave 1984) 
is exploring the relationships among cog- 
nition, culture, and context and makes a 
similar point: that learning in school-like 
settings may need to be reorganized or 
rethought in line with what is being 
learned about learning in daily life. 



Programs as Settings 

Most adults in the United States seeking 
literacy instruction have attended school, 
with many having completed several years 
of high school. Adult literacy programs 
differ dramatically in the extent to which 
they replicate the routines of prior school- 
ing or establish new roles and expecta- 
tions for learners and teachers. Some 
programs (and learners) may view the 



purpose of teaching and learning as 
remedying school failure and thus may 
concentrate the curriculum on school 
learning narrowly defined. Others may 
see their role as traming adults in task- 
specific literacy fikills, such as those 
required in particular workplaces. Still 
others may view teaching and learning as 
contributing to broader notions of indi- 
vidual and community development and 
may see the roles of teachers and learners 
as more fluid and interdependent. Some 
programs emphasize more than others the 
knowledge and experience learners bring 
and expect their successes as learners in 
other domains to contribute to their 
growth in literacy. 

Adult literacy programs have been charac- 
terized in the literature according to 
different features and emphases. Hunter 
and Harman (1979) distinguish between 
conventional adult literacy programs that 
share the primary goal of reading develop- 
ment and functional programs that con- 
centrate on the development of particular 
competencies or on the "functional con- 
cerns of ethnic or cultural subgroups" 
(p. 59). Following Hunter and Harman, 
others have made distinctions between 
program types according to their orienta- 
tion and to their approaches to literacy 
education (Association for Communi^ 
Based Education 1986; Fingeret 1984). 

Fingeret (1984) categorizes U.S. literacy 
programs as either individuaUy or com- 
mimity oriented. According to Fingeret, 
most programs are individually oriented; 
they focus on individuals' acquisition of 
literacy skills with the primaiy goal of 
mamstreammg into middle-class society. 
Community-oriented programs, on the 
other hand, are based on other assump- 
tions. Offering literacy as one of a 
number of services geared to the needs of 
specific groups or conmiunities, these 
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programs often reach those most econom- 
ically poor and in need, those not served 
in large numbers by adult basic education. 
Individually oriented prograi^ stress 
individual empowerment, according to 
Fingeret, whereas conmiunityK)riented 
programs emphasize collective develop- 
ment and politicized action. 

Few programs fit all the features 
attributed ejt>cr ot these program 
orientations. As Fingeret mentions, 
individually and community-oriented 
programs stare a number of features and 
often have the qualities of one in the 
other. A multisite liieracy program 
operated out of a school district, for 
example, might have within it classes and 
teachers resembling a community-oriented 
model, at the same time operating within 
the accountability system and implicit 
philosophical stance of the bureaucratic 
organization of which it is a part. 
Depending on the stance of the analyst, a 
particular program may be defined both 
as individually and community oriented. 
^Mien programs participate in discussions 
about program types, they may uncover 
components in their own program that are 
contradictory or incoiigruent. 

A growing number of literacy programs 
have been directed at literacy skills for 
the work force. Literacy services to 
employees have most often been provided 
throu^ relationships with local school 
boarc^ and cc omunity colleges. How- 
ever, the past few years have seen the 
development of literacy programs inte- 
grated with work settings, creating the 
potential for curriculum and instruction to 
be derived from the literacy practices of 
the workplace (Business Council for 
Effective Literacy 1987). There has also 
been an increase in programs offered in 
workplaces by service providers; these 
may or may not relate directly to the on- 



the-job litei'acy needs of participants. As 
workplaces increasingly become the set- 
tings for liteiacy learning, issues about the 
breadth or niurowness of the curriculum, 
about the corgruence between the goals 
of learners anc^ their employers, about the 
effects on thost unable to participate, and 
about the relationship between €»nhanc:d 
literacy and job .nobility are being raised. 



Cultures of Teach.\ng 

Although there is currently a heightened 
awareness in the United States about 
adult literacy leamei'S and services, com- 
paratively little attention has thus far been 
focused on the teachers and tutors work- 
ing in those programs. Few in number 
when compared to the teachers working in 
primary and secondaiy education, these 
teachers and tutors, paiJ and volunteer, 
full time and part time, have been largely 
ignored in national or local calls for 
educational reform, formal teacher train- 
ing, or staff development. 

In part because adult literacy programs 
are on the margins of the educational 
system, the quality of teaching in them 
varies considerably. Many experienced 
instructors have been isolated from 
developments in the field such as new 
approaches to teaching and advances in 
research or theoiy. Others teach in 
programs part time after long days as 
primary or secondary school teachers, and 
still others teach in a literacy program for 
a year or two and tiien leave the field. 
Volimteers often have full-time jobs and 
can make only a limited commitment to 
their tutoring. At issue here is the degree 
to which the field should be professional- 
ized and what knowledge and experience 
are important in teaching literacy. 
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I'eachers and tutors often bring rich and 
varied experiences to their work, whether 
they have had any prior training specific- 
ally related to the teaching of literacy or 
adults. Many of these instructors have 
taken nontraditional routes into teaching, 
coming from experience and employment 
in human services, industry, and commun- 
ity organizing. Others have themselves 
been adult learners in literacy programs. 
Inere is considerable debate about the 
advantages and disadvantages of formal 
education and certification for this cohort 
of tea:hers, as well as about what sorts of 
training and orientation are realistic and 
appropriate for volunteers. 

Adult literacy programs as workplaces 
exhibit some of the same problems as 
schools. Structures for collaborative 
planning and teaching are difficult to 
♦ablish. Many adult literacy teachers 
. uoiij^'meet with their classes cr tutees 
in decentralized sites where they may 
rarely, if ever, see their co)le?.gues. Their 
status as part-time worker ^ creates dis- 
incentives to expand their responsibilities. 
Few opportunities exist to observe other 
instructors or to meet with teachers and 
learners in informal ways that play an 
important role in building a program 
community. Like elementary and second- 
ary schools, literacy programs may need 
their own version of restructuring and 
reallocation of resources in order to 
further the professionalization and 
stability of their staff. 

Community-orieuted programs may expe- 
rience some of these problems to a lesser 
extent although they also have their own 
concerns. To function effectively in these 
settings, teachers need community knowl- 
edge, background, and experience, as well 
as training as facilitators and advocates 
(Fingeret 1984). Sometimes programs are 
faced with helping staff to Kmleam their 



formal training" and adopt new behaviors 
and expectations appropriate to the set- 
ting, like teachers in other types of 
programs, however, educators m 
communi^-oriented programs need oppor- 
tunities for thoughtful reflection about 
their teaching and ac^^ss to wider net- 
works of Ideas and practices. 

Another debate in adult literacy programs 
involves conflicting orientations toward 
the roles of teachers and learners. Some 
regard the role of teachers of adults as 
primarily diagnostic: using expertise to 
assess skiU deficiencies and prescribing 
appropriate sequences of instruction. 
Brooldield (1985), on the other hand, sees 
teachers as the facilitftors of adult 
learning, the negotiators of goals, curric- 
ula, and evaluative criteria. He challenges 
the view of adult education where &e 
instructor's role is to accept uncritically 
learners* perceptions of their own learning 
needs and design education to meet these 
needs. He stresses that this is an 
improvement over past models where sole 
authority for learning goals rested with the 
instructor; nonetheless, this "new ortho- 
doxy" is potentially "apolitical" and 
"acurricular." 

Brookfield's view is reminiscent of Freire's 
(1972) notion of praxis^ in which teachers 
and adult learners engage in continuous 
and alternating processes of action and 
reflection. Knowledge is not fixed, adults 
learn as much firom peers as fi-om instruc- 
tors, and a premium is placed on problem 
posing, risk taking and learning to deal 
with questions and ambiguity. Auerbach 
and Wallerstein (1987), in a worlq)lace 
example of this model, center their teach- 
ing approach on the "shared nature" of the 
"conflicts and problematic interactions 
which enable studentC' to envision different 
working conditions and fashion an individ- 
ual or community response to the 
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problem" (p. 2). From this perspective, 
teaching becomes a deliberative and 
leflective activity (Dewey 1916; Schon 
1983; Schwab 1969). In the cuixenl litera- 
ture on teacher education and staff devel- 
opment (for exaiiple, Zumwalt 1982), the 
view of teaching as a reflective, intellect- 
ual activity is often contrasted with the 
technological view of teaching as com- 
posed of a definable repertoire of knowl- 
edge, skills, and attitudes. 

>\lthough these images of teaching and 
learning may be discernible in particular 
programs, the picture is at the same time 
more complex than dichotomous models 
would suggest. A teacher may on differ- 
ent occasions shift from one perspective to 
another. Teachers in the same program 
may hold quite different views. The 
curriculum of the program itself may 
reflect a range of philosophir»! .^ances. 

The issues mentioned in this section on 
educational contexts raise a numbfir of 
questions relevant to the evaluation of 
programs and the assessment of learners. 
If much significant learning occurs outside 
of programs, and if the nature of that 
learning differs from formal education, 
what then are the implications for creating 
learning environments more congruent 
witli adults' daily experiences and for 
assessing that learning beyond program 
boundaries? Since programs differ so 
dramatically from each other, particularly 
in their relationship to specific community 
• projects or work force requi rements, stra- 
tegies tor program evaluation will need to 



respect these differences while still provid- 
ing some bases for comparison. Describ- 
ing the cultures of teaching and tutoring 
in adult literacy programs raises questions 
about ways to create the incentives and 
opportunities for teachers and tutors of 
adults to be learners themselves, to have 
professional development experiences that 
involve collaboration, reflection on 
practice, and access to theory and 
research. 



Conclusion 

This chapter has provided an overview of 
contrasting perspectives on adults, literacy, 
and educational contexts in order to pro- 
vide starting places for formulating ques- 
tions about program assumptions, pro- 
cesses, and outcomes. Many of these 
issues are both difficult and delicate, for 
they invoke inquiry into deeply held 
beliefs and habitual practices. Program 
evaluadon and learner assessment are 
more than technical matters, and so 
require thoughtful discussion of a wide 
range of program concerns. In the next 
chapter are reviewed an array of publica- 
tions related specifically to the design and 
implementation of program evaluations in 
different contexts. Implicit in these mate* 
rials are particular views of learners, 
literacy, and educational contexts that 
demonstrate choices made by program 
staff and other stakeholders. The figure 
that follows suggests a few of the ques- 
tions about adults, literacy, and educa- 
tional contexts with implications for the 
design of evaluation and assessment. 
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Adults as Learners 

o How is the program designed specifically for adult learners? 

o How congruent are teachers'/tutors' and leamei-s* concepts of success? 

0 What roles do adult learners and staff play in setting and r—'iing program 
goals? 



Concepts of Literacy 

o What is the program's working definition of literacy? 

0 What information about the cultures, communities, and expectations of adult 
learners has been used to arrive at this definition? 



Educational Contexts 

o To what extent does the design of the program relate to adult learning in 
eveiyday life? 

o Is the program more individually or community oriented? 

o How does the program provide opportunities for teachers and tutors to 
expand their instructional repertoires? x. 



Figure 1. Contrasting perspectives on auult literacy education: 
some questions for programs. 
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RESOURCES FOR PLANNING 
PROGRAM EVALUATIONS 



Introduction 

The purpose of this chapter is to review 
critically the available literature related to 
program evaluation in adult literacy edu- 
cation^ showing what the pattern of work 
has been to date and suggesting strengths, 
limitations, and omissions that have impli- 
cations for the further design and imple- 
mentation of program evaluation in the 
field. 

Current research as well as practical 
expt»,rience r'lggest th^t program evalua- 
tion can serve many purposes. An effec- 
tive system of documentation and moni- 
toring provides information to address 
questions raised by program staff and 
enables staff and administrators to plan, 
acquire, and maintain funding and in 
some cases to seek approval as a nation- 
ally validated model. Evaluation may 
help to {^certain if a program or group of 
programs merits receiving or continuing to 
receive financial support from a funder, or 
to justify the ^location of resources within 
a program. Critical syntheses of program 
evaluations, perhaps carried out by 
umbrella organizations, research centers, 
or other knowledgt^able groups, may pro- 
vide mechanisms for disseminating pro- 
mising practices across program sites and 
encouraging networking. 

No one system or approach to program 
evaluation is likely to accomplish such a 
wide range of purposes and serve so many 



different audiences. In fact, the 
tremendous diversity in adult learners, 
sites, programs, and goals makes com- 
parability across some programs inappro- 
priate and argues for diversity in 
approaches to program evaluation. As 
Cuba and Lincohi (1981) point out, the 
worth of a program or practice relates to 
its value ia a particular context. Eval- 
uation designs, therefore, need to be true 
to the nature of the programs for which 
they have been developed. 

Raising the awareness of program mana- 
gers and staff about promising practices in 
evaluation is an important initial step in 
enhancing program quality. Based on 
their prior experience, som practitioners 
may equati program evaluation only with 
collecting scores and data on attendance 
patterns, and they accept those statistics as 
evidence of program success or failure. 
Others view those statistics as reductive, 
inappropriate to their program processes 
and goals, and even potentially threaten- 
ing, depending on how the information is. 
interpreted and used. Policy makers and 
funders, on the other hand, often have 
little access to evaluative data &bout 
programs apart from scores and demo- 
graphic information, and thus little else 
apart from quantitative data on which to 
base critical decisions. All would, there- 
fore, benefit from greater knowledge of 
evaluation. 
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A search of the literature related to pro- 
gram evaluation in adult literacy educa- 
tion yields a wide range of documents, 
only a few of which are actual examples 
of evaluations of particular programs. 
Many other reports, studies, and mono- 
graphs, however, contain relevant and 
useful information, and so have been 
categorized in this review. These include 
program surveys, handbooks, instruments, 
and policy studies, as well as designs for 
multisite data collection leading to 
assessment and evaluation. Although 
many programs prepare annual reports for 
boards and funders, these have not been 
collected and analyzed here as resources 
for or examples of program evaluation. In 
addition, although every effort has been 
made at comprehensiveness within each 
category, there may be inadvertent omis- 
sions because some of this material is not 
accessible in libraries or through database 
searches. Each section characterizes the 
type of literature, describes the available 
documents of ihis type-their purposes, 
methods, and selected findings-and sug- 
gests how they might be useful m plannmg 
program evaluations. 



Program Surveys 

Conducted for a range of purposes and 
audiences, surveys of programs in adult 
literacy provide descriptive information 
about seivice provision. Survey methods 
include site visits, interviews, mail 
questionnaires, and audio recording of 
meetings; they generally conclude with 
some recommendations for further 
reset^rch, development, and/or technical 
assistance. Some surveys identify distin- 
guishing features of broad types of pro- 
grams and/or patterns of practice in geo- 
graphical areas. In their design and find- 
ings, surveys often suggest the different 
meanings that "literacy** has assumed in 



different educational contexts. Surveys 
also provide information, often from prac- 
titioners' perspectives, about program 
goals iind components and about how 
resources have been organized to facilitate 
a particular approach to teaching and 
learning. Thus, programs planning eval- 
uations might consult a survey of similar 
programs to consider and raise questions 
about differences or invent other uses for 
program surveys beyond those originally 
intended. 

Surveys of adult literacy programs have 
different purposes. Some surveys are 
' conducted to synthesize information abcut 
programs of a particular type (for exam- 
ple, conununity- or industry-based pro- 
grams) or to collect information about 
programs in a particular geographical area 
(such as a ci^ or state). Some surveys 
provide detailed, program-specific infor- 
mation using a standardized set of 
descriptors across a large sample of 
programs in the category of that ^ecific 
type (Association for Obnmiunity Based 
Education [ACBE] 1986), whereas others 
synthesize and critique program data with- 
out differentiating particular programs 
(Chall and Heron 1986). Some describe 
in detail selected cases or examples 
(Fields, Hull, and Sechler 1987). 

Some surveys are particularly useful to the 
field because they analyze and synthesize 
hard-to-obtain data. The study carried 
out by ACBEX1986) of community-based 
literacy programs is an example of this 
type. ACBE set out to provide a compre- 
hensive description of programs &at, 
m<my believe, represent a major direction 
in adult literacy education (Fingeret 
1984). The goal was to describe the 
philosophy and practices of programs that 
linked education and conununity 
development in oroer to meet the needs 
of disadvantaged constituencies. (See 
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Fingeret 1985 for a description of 
community-oriented literacy programs.) 
The results were intended to strengthen 
already existing programs and to provide 
guidelines for ii\dividuals and organiza- 
tions interested in st;^ting new ones. In 
this survey, ACBE identified a number oi 
community-based programs around the 
country rjid described some of * their 
"unique and outstanding** characteristics, 
approaches, and problems. These charac- 
teristics include work with particular 
constituencies, institutional independence, 
focus on economic and social, self- 
sufficiency, and individual and community 
empowerment. The survey report com- 
pares ACBE's assumptions about 
community-based programs with what they 
learned from the field. Through this 
survey, ACBE discovered that the use of 
learner-centered, participatory methods 
characterized many, but not all, 
community-based programs. Finally, 
ACBE learned that community-based pro- 
grams have an array of outcomes but "few 
have time or resources to rigorously 
document results that occur" (ACBE 1986, 
p. 56). 

Similar in purpose to the ACBE study but 
focused on a different type of literacy 
program, a survey of industry-based pro- 
grams (Fields, Hull, and Sechler ' 987) set 
out to identify and describe efforts 
designed for basic skills instruction in 
industry and to identify trends in their 
practices and approaches. Both surveys 
reafSrm the notion that no one system or 
process of program evaluation is likely to 
be suitable across vastly different 
programs. 

Surveys of programs based on location 
often have an explicit agenda to examine 
instructional and policy issues in order to 
inform government agencies, funders, or 
other organizations such as colleges or 



universities (Chall and Heron 1986). 
Gadsden (1988), for example, examined 
the characteristics of adult literacy 
programs in the state of Michigan by 
surveying directors and coord^iators, 
making site visits to develop a survey 
instrument, and designing and implement- 
ing a mail survey. Whereas the programs 
in Michigan cite recruitment of volunteers 
as their major problem, the d'c Si lead 
outsiders to conclude that an important 
concern may be finding strategies to 
attract more participants. Gadsden sug- 
gests that new methods of advertising, 
marketing, and providing services are 
needed; people and programs need to be 
'Svilling to go into conmiunities and use 
community resources and services to 
attract students" (p. 161). 

Designed to inform literacy policy in 
California, a recent survey conducted by 
Solorzano, Stecher, and Perez (1989) for 
the California State Department of Edu- 
cation consisted of an extensive review of 
the literature, a nationwide telephone 
survey of 63 exemplasy programs, and 4 
site visits. Researchers gathered data 
about a wide rar •'.e of program features, 
including inst onal approaches, staff 
development, Si ategies for reaching the 
"hard to reach," grouping practices, types 
of written curriculum, fi-equency of curric- 
ulum revision, differentiation of curric- 
ulum according to ethnicity and language, 
and leadership qualities of managers. 
Programs reported that their evaluations 
are based on grade-level increases, GED 
passing rates, retention rates, posttest 
scores, mastery of competencies, course 
completion rates, enroUment increases, 
and changes in self-concept Sixty-three 
percent of the programs had l)een for- 
mally evaluated, although only 16 percent 
have a budget for evaluation. 
Community-based organizations were 
found to have the highest retention rates. 
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The report urges programs to see evalua- 
tion as a tool, not a threat. Recommen- 
dations to the state of California include 
coordination of services among agencies; 
greater sensitivity to the needs of both 
urban and rural communities; shared deci- 
sion making among administrators, stafi, 
and learners; provision for support ser- 
vices beyond literacy; jnd a de-^mphasis 
on tests or any particular instructional 
approach. Surveys such as this one are 
designed to influence policy, in this case 
at the state level, and thus may also be 
considered a form of policy research. 



Handbooks 

In the field of adult literacy education, 
handbooks and manuals have been devel- 
oped for a wide range of purpose*", and 
audiences. They are included in this 
chapter because of the questions they 
pose and because the components they 
suggest for program development can 
provide categories and concepts for the 
design of evaluations. Not included are 
handbooks written primarily for a local 
audience that usually provide some over- 
view of the field as a prelude to describ- 
mg, often eclectically, several approaches 
to instruction. Written to educate staff; 
instructors, and volunteers new to the 
field, these handbooks are how-to books 
that do not typically use theory or 
research in their discussions nor attempt 
to deal with complex issues in the field 
(Fingeret 1985). 

More broadly based handbooks are often 
developed from the type of program sur- 
vey described earlier and may include 
some synthesis of the research literature 
as well. In general, these fall into two 
types: (1) handbooks intended for literacy 
programs generally (Lerche 1985b; Mayer 
1989) and (2) handbooks focused on a 



particular type of literacy program, such 
as workplace programs (Business Council 
for Effective Literacy 1987), programs for 
the elderly (Jacobs and Ventura-Merkel 
1986), or participatory programs (Jurmo 
1988). All of these handbooks aim to 
provide descriptive, state-of-the-art 
information of immediate practical value 
and to inform program designers and staff 
of the critical activiti'';;s and considerations 
involved in setting up and administering 
literacy programs of various types. 

Most handbooks have ^jeen developed 
through a process of suivey research 
and/or field involvement and input. 
Lerche (1985a) provides a practitioner- 
oriented synthesis of the findings of the 
National Adult Literacy Project (NAUP), 
a government-funded study of how literacy 
programs structure program operation and 
management to meet their objectives. 
Drawing on a mail survey of more than 
200 "exemplary" programs as well as in- 
depth interviews with staff and students of 
32 of these programs, the study yielded 
descriptive data about how programs in a 
wide variety of settings formulate, imple- 
ment, and evaluate plans for adult literacy 
education. Project director of NALP, 
Lerche designed the survey report as a 
ht ndbook for practitioners that uses the 
survey data as "state-of-the-art" informa- 
tion to offer guidance in initiatmg, modify- 
ing, and improving program design and 
operation. What the handbook does not 
include is current information from other 
sources, that is, theoiy and research in 
literacy education more generally. The 
extent to which the surveyed practices are 
congruent or incongruent with current 
theory is thus not addressed. 

Handbooks designed for literacy programs 
in general assume that many programs in 
this country have rather similar 
components. For example. Guidelines for 
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Effective Literacy Programs, prepaicd by 
Mayer (1989) for B. Dalton's National Lit- 
eracy Initiative, includes guidelines for 
relations with the community, stafit 
resources, instruction, governance, man- 
agement, and evaluation. The guidelines 
are intended to be used as a framework 
for program self-a:.sessment. According to 
the au2ior, discuss'o.n of these guidelines 
can help program staff and boards identii^ 
consensus and di:.agreements about pro- 
gram directions, provide inservice training, 
and communicate with funders and/or 
make grant requests. The NALP study 
(Lerche 1985a) describes a similar set of 
program components. Descriptions such 
as these may be used intemaJly to iden- 
tify questions and areas for evaluation. In 
addition, these handbooks provide exter- 
nal evaluators who have limited experi- 
ence in the adult literacy education field 
with "catalogues** of how programs tend or 
intend to work. 

Other handbooks are more clearly linked 
to specific philosophies or missions in the 
field. Jacobs and Ventura-Merkel (1986) 
provide a handbook for organizing a lit- 
eracy program for older adults using a 
model in which older adults provide lit- 
eracy education to their peers in conmiun- 
ity locations they frequent. The ideas are 
intended to be used by any organization 
concerned with the literacy needs of older 
Americans. Another handbook (Jurmo 
1988) is oriented exclusively to participa- 
tory literacy education, and more specific- 
ally to participatory approaches to instruc- 
tion and management. Jurmo suggests, 
for both professionals and volunteers, a 
"process by which they examine their own 
assumptions about participatory education, 
study what others have already done, and 
then collectively build a system for devel- 
oping participatory practices within their 
own programs" (p. iv). The handbook 
proposes a set of steps through which 



i ^actitioners can critically examine iiieir 
own practices in relation to the efforts of 
others. 

Several of the handbooks include sections 
explicitly devoted to program evaluation. 
The most extensive discussion is provided 
by Lerche (1985b), who diaws on NALP 
data to describe reasons for conducting 
program evaluation, barriers to using 
sophisticated evaluatior. methods, and 
procedures for evaluation. In internal 
evaluations, Lerche points out, the direc- 
tor, program sta^ and learners all can 
play a role. Four recommendations are 
made: (1) developing evaluation expertise 
through consultatioii with local evaluation 
experts; (2) df<fining program goals and 
objectives by eidistlng the participation of 
learners, staff, and community members; 
(3) designing both formative and summa- 
tive evaluation instruments; and (4) creat- 
ing a unified system for data collection, 
analysis, and use. 

Handbooks often contain ideas for orga- 
nizing program evaluations. Mayer (1989) 
recommends an annual program review 
under the direction of the board of direc- 
tors or an advisory committee. Such a 
review would include questions developed 
in consultation with learners, instructors, 
staff, board members, and key community 
constituents. Each section of the B. 
Dalton guide%es may be treated as a 
checklist of elements to consider in the 
review process. 

Both the B. Dalton and NALP handbooks, 
designed for general use iii the field, 
argue that program evaluation should be 
consistent with the program's mission, 
philosophy, and expected results. Because 
they begin with this assumption-that pro- 
gram evaluation reflects the individuality 
of particular program contexts-neither 
provides any criteria for selecting which 
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data to collect or bow to analyze aud 
it l«.rpret them once collected. Both 
handbooks assume that the program goals 
provide the framework wiUiin which the 
other components are evaluated. Al- 
though this position both reflects and 
respects the diversity of the field, it does 
not acknowledge the value of questioning 
the goals themselves (Fingeret 1984). 



Instruments 

Just a few examples of instruments for use 
in program self-evaluation are currently 
available in the literature (Lerche 1985b; 
Willing 1989). . Arguing for well-planned, 
broad-based, and systematic efforts at 
program evaluation at the local level, 
WiUmg draws on the self-study processes 
institutionalized by regional accreditation 
bodies to design a sample evaluation tool 
for adaptation by individual programs. 
Eight program components-administra- 
tion, planning, instruction, staff develop- 
ment, community involvement and pubhc 
relations, evaluation, student services, and 
finances-are broken down further into 
elements with two to Tjut standards writ- 
ten for each. The standards consist of 
indicators or criteria with rating scales to 
ascertain the extent to which the program 
standard is being met. The instrument is 
intended to be used as a starting point for 
programs to develop their own locally 
appropriate process. 

Another instrument has been developed 
by the Division of Adult and Community 
Education of the Indiana Department of 
Public Education and piloted by the 
Lafayette Reading Academy (as described 
and reprinted in Lerche 1985b). Com- 
pleted by the program administrator, the 
first part collects qualitative and quan- 
titative data about goals, enroUment, 
seasonal attendance patterns, number of 



program sites, and a calculation of pro- 
gram impact based on a formula relating 
number of students attending to reasons 
for separation from the progranL The 
second part of the self-evaluation is 
completed individually by program staff 
members and returned to the program 
administrator. It asks staff to respond to 
questions about nine areas of program 
operation, including perceptions about 
program direction, administration, staffing, 
facihties, instructional materials rjid 
processes, and assessment. The docur^.'^.nt 
is intended to be used as is < lot adapted), 
and although it ir'^ludes a vide range of 
types of items (open-ended questions, 
scales, checklists, and so forth), it does not 
include instructions for scoring and inter- 
preting the data. 

These two instruments are similar in sev- 
eral respects. They assume that a paper- 
pencil survey is a central process in eval- 
uating literacy programs, that statistics will 
capture some but not all of the relevant 
data, and that program self-evaluation can 
be an informative and useful process for 
program planning and development. Al- 
though these instruments involve both 
adnunistrators and staff in the process, 
neither solicits data directly from program 
participants or involves them in the design 
or revision of the evaluation instrument. 
Neither instrument explicitly acknowledges 
that differences in fundamental purposes 
and contexts for literacy programs may 
require the programs to use different com- 
ponents and standards. 



Policy Studies 

For the purpose of this review, the cate- 
gory of policy studies includes survey 
evaluations of literacy programs at the 
national, state, and local level designed to 
address broad issues related to compliance 



24 



with federal legislation, the extent anrl 
type of need for adult education services, 
features of progi am participation, and a 
range of program processes and impacts. 
For policy studies, then, the audience is 
more general than for the evaluatiom> of 
particular programs intended to contribat^. 
directly to the decision-making processed 
of local stakeholders. Primarily intended 
to inform policy makers and federal and 
state officials responsible for program 
mom'toring and management, these studies 
are also of interest to practitioners at the 
state and local level, because they place 
the praaices of individual programs within 
n broader poli:^ context. The goals, 
m-^thods, and recommendations of policy 
studies at the national and state level thus 
have the potential to provide a valuable 
resource to the design of program evalua- 
tion at the local level. 

Several studies have IwSn conducted at 
the national level to assess the state- 
administered programs of the Adult Edu- 
cation Act (AEA). A study conducted for 
the OfGce of Program Planning and Eval- 
uation of the U.S. Department of Educa- 
tion (Young et al. 1980), for example, was 
intended to iSll a national gap in coherent 
data on local program operations. By col- 
lecting suivey data from state project 
directors, making site visits to gather 
further data about and from teachers and 
program participants, and interviewing 
representatives of local agencies and 
organizations, the study provides an ana- 
lytic description of the state programs 
across the nation and suggests a wide 
range of program characteristics and 
impacts that could be the focus of further 
investigations. In addition to such 
noninstructions ^ characteristics of pro- 
grams as staffing and enrollment, financial 
resources and expenditures, and suppor- 
tive services, the evaluators looked at 
instruction (teacher characteristics, course 



content, teaching strategies and emphases, 
settings for instruction) and program prac- 
tices from the perspectives of participants 
(outreach and recruitment, early with- 
drawal, goals and goal attainment, use of 
auxiliary services, and achievement and 
satisfaction related to teacher training and 
style). Eighty-one percent of adult educ& 
tion programs nationaUy were found to be 
admhiistered by local school districts, 12 
percent by community colleges, and the 
rest by vocational institutes, educational 
service agencies, and comn'iUnity-based 
organizations. In contrast to the Con- 
gressional emphasis on educating adults to 
enhance economic independence, this 
study indicates that state and local 
administrators emphasize educational and 
personal development goals and benefits 
more than go^ and benefits directly 
related to employment 

Also at the national level, Moore and 
Jung (1985) use already available state- 
collected data to review need and impact 
in adult education in 12 states. In tiieir 
review of previous studies, the authors 
pull together findings from five major 
formally commissioned investigations of 
adult education programs from 1976-1981 
(including the study by Young et al. 1980). 
They conclude that (1) the need for adult 
education services considerably exceeds 
federal program resources; (2) the number 
of adults participatmg nationally has 
increased considerably since 1985; 
(3) adult education nationally is provided 
in three types of programs~ABE, adult 
secondary education, and ESL; (4) there 
are particular problems defining and tar- 
geting resources to those '*most in need"; 
and (5) progr mis rely primarily on exist- 
ing educatioral institutions. Although the 
quality of data collected on participant 
outcomes was found to vary substantially 
across states, causing problems in consis- 
tency, comparability, and accuracy, the 
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range of types of data collected was con- 
siderable. Notable among outcomes 
related to academic achievement and 
social accomplishments were programs 
collecting data on participants' knowledge 
of government and law, com munity ser- 
vices, parentmg skills, economics, health 
care, income tax preparation, and occupa- 
tions. Part of this report focuses on 
literacy efforts staffed by volunteers and 
those using existing instructional and 
conmiunity-based organizations. Major 
differences were found among state pro- 
grams in their usage patterns of volunteer 
aT^d private sector resources across the 
states. 

In addition to nationally commissioned 
surveys of state and local programs, many 
states work with evaluation consultants to 
do extensive surveys of the range of pro- 
grams funded through the ABA and state 
monies (Bonnet and Elston 1988; Hughes 
and Braunon 1988; Sherron 1986; and 
Snow and Bentley 1988). These evalua- 
tions are designed to assist program 
managers in making decisions and to pro- 
vide data for the management of adult 
education at the state and federal levels, 
intended to provide basic descriptive 
information, most state studies use pri- 
marily mail and telephone surveys of pro- 
gram directors, staff, and sometimes adult 
learners. They include a range of pro- 
gram features such as recruitment and 
intake, planning, characteristics of staff 
and participants, scheduling of services, 
curriculum, instruction, and asses sment 

In contrast to studies relying pi 1 warily on 
survey data, the state of Kentucky has 
developed and piloted an innovative sys- 
tem for evaluating ABE programs within 
the state (Morehead State University 
1987). The system involves ongoing peer 
evaluiition using instruments developed by 
teacher*; and by the Illinois State Board of 



Education. After training in the philos- 
ophy of the system, 10 adult educators led 
25 evaluation teams in the evaluation of 
11 adult education programs throughout 
the state. Similar to the evaluation pro- 
cess used in other state programs such as 
teacher and adnunistrative certification, 
this process has advantages to both eval- 
uators and programs in effecting program 
improvements. 

In their comprehensive review of national, 
state, and local studies on the outcomes of 
participation in ABE, Darkenwald and 
Valentine (1984) conclude that although 
results have not been consistent, many 
studies have emphasized change in self- 
concept as the most dramatic result. Also 
reported are improvements in reading and 
writing and some gains in employment for 
ABE particip; nts. The authors point out, 
however, that the majority of participants 
enrolled for reasons of education rather 
than employment, a finding that raises 
questions about the current national 
emphasis on literacy education for 
employment and the extent to which 
workplace-related or job training pro- 
grams can meet broader educational 
needs. Critical of the quality of the 
available research, Darkenwald and 
Valentine argue that more research is 
needed to identify the diverse outcomes 
for the individual participating in ABE. 
Their work suggests the need for policy 
research in basic literacy education that 
provides comprehensive data about rea- 
sons for enrolling in relation to learner 
outcomes. 

Some studies of participation in ABE 
suggest outcomes that could be incor- 
porated into the design of program eval- 
uations. In their own study, Darkenwald 
and Valentine (1984) set out to determine 
the costs and benefits of New Jerse/s 
high school completion programs, to 
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consider the impact of these programs in 
terms of student goal attainment and tan- 
gible indicators of social and economic 
well-being, and to design a statewide 
model for follow-up. Using telephone 
interviews and questionnaires, the study 
gathered participants' self-reports about 
employment, basic skills, self-image, 
participation in further education, and 
even parenting to provide persuasive 
empirical justification for continued or 
increased investment in the state s ABE 
programs. Another study, conducted in 
Iowa by Beder and Valentine (1987), was 
intended to improve ABE recruiting pro- 
cedures by providing a comprehensive, 
descriptive analysis of the Iowa ABE 
population based on a carefully drawn 
random sample. As a policy research 
project, it was intended to help the state 
education department determine partici- 
pant satisfaction anjl more effective 
recruiting methods. 

Several statewide studies have been 
designed to inform polity making. The 
California Literacy Campaign Program 
Effectiveness Reviews (Line et al. 1984; 
Wurzbacher and Yeannakis 1986) used 
document analysis, telephone surveys, and 
written questionnaires to de scribe program 
participants and identify outcomes of ser- 
vice. A few policy studies have focused 
on the implementation of particular prac- 
tices, as in Solorzano and Stecher's (1987) 
survey conducted on behalf of the Educa- 
tional Testing Service of the methods cur- 
- rently used by the 46 Califomii literacy 
Campaign sites to assess learner progress. 
Evaluators collected data from library 
staff, distributed a written questionnaire 
and visited sites. The findings provided 
evidence that strategies for assessing 
learner progress were being used but that 
new ones were clearly needed. 



In the first phase of a longitudinal smdy 
recently conducted in New York Qty 
(Denny, Albert, and Manes 1989), 
researchers at the literacy Assistance 
Center gathered data during 1987-1988 
about the demographic characteristics of 
adult students in basic education and 
English for Speakers of Other Languages 
(ESOL) programs, their educational and 
family literacy backgrounds, their past and 
present participation in the labor force 
and public assistance programs, as well as 
their employment aspirations. Informa- 
tion was also solicited about students' 
motivation for attending programs and the 
types of accomplishments and levels of 
satisfaction students experience. Data 
were collected through individual inter- 
views wiUi 663 students and a n 'led 
survey to program managers at 38 adult 
literacy program sites. Among the issues 
identified for further research and discus- 
sion were students' lack of awareness of 
nonliteracy services available and the 
delicate balance between recruiting stu- 
dents and having adequate places avail- 
able for them to enroll. A large number 
of students in the study cited the "less 
tangible" results of participation such as 
increased self-confidence and "a better 
life," as Darkenwald and Valentine (1984) 
also found, lending strength to the need 
for additional ways to assess progress 
beyond standardized tests of basic slulls. 

The poliq'/cvaluation research on impact 
of programs in ABE and adult literacy 
suggests a number of issues that merit 
further consideration. The first is that a 
range of program outcomes needs to be 
considered, although some may not be 
clearly specified or intended in the pro- 
gram goals. Furthermore, as Fingeret 
(1985) pcints out, evaluation studies in 
adult literacy that aim for understanding 
of instructional program dynamics and the 
perspectives of participants 3'e very much 
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needed. In addition to outcomes or 
impact, such studies would explore the 
complex interactions among students, pro- 
grams, and the context or environment 



Literacy Program Evaluations 

In this section, four comprehensive reports 
are described in detail to illustrate dif- 
ferent approaches to and purposes for 
evaluating adult literacy pro-ams. Th^ 
examples show how formative and sun 
tive evaluation of program implementation 
and impact may contribute to the develop- 
ment of individual programs as well as to 
the knowledge base in adult literacy 
education. In formative evaluation, the 
evaluator describes and tries to under- 
stand how the program is developing, the 
"dynamic pro-am processes and their 
holistic effects on participants'* (Patton 
1987, p. 18). Primarily qualitative- 
naturalistic, these evaluations try to 
capture in detaQ the quality of the 
program and its strengths and weaknesses, 
and thus pay considerable attention to the 
perceptions of program staff and learners 
about what is going on. Summative eval- 
uations are designed to "make basic deci- 
sions about whether a program is effective 
and whether it should be continued" 
(Patton 1987, p. 28). Summative evalua- 
tors typically report to a funding agency, 
government ofQce, or program advisory 
board. Evaluation reports include descrip- 
tions of the program, achievement of pro- 
gram goals, unanticipated or related out- 
comes, and sometimes comparisons with 
other programs. Some evaluations are 
both formative and summative (Hikes 
1988a; Koen 1986b; Koen and Musumeci 
1984; Turner and Stockdill 1987). An 
evaluation of program implementation 
describes how the program operates, 
whereas an evaluation of impact shows 



the results of its activities and elEfects on 
learners. 



Literacy Volunteers of 
New York City 

Given the priorities and financial con- 
straints faced by most adult literacy 
programs, it is not surprismg that the 
literature contains fe v examples of 
program-initiated, systematic evaluation 
over time. The three successive studies of 
Literacy Volunteers of New York City 
(LVNYC) conducted by Matrices, Inc. in 
1981-82, 1983-84, and 1985-86 (Koen 
1986b; Koen and Musumeci 1984), are 
based on a gradually emer,'png set of 
related questions, and provide an example 
of external program evaluation designed 
with input from program staff. The 
studies investigated key questions related 
to both program implementation and 
impact, and in particular to instructional 
issues aroimd methods and groupmg pat- 
terns. The first study (1981-1982) raised 
a question about the impact of instruction, 
and more specifically, whether LVNYC 
program participants reached a plateau 
after SO hours of instruction. The second 
(1983-1984) looked at internal consistency: 
the extent to which the instructional pro- 
cesses used in tutoring sessions were con- 
sistent with LVNYCs training program 
for tutors and its instructional phUosophy. 
Findings showed how instructional time in 
individual and group sessions was being 
spent and depicted the respective roles of 
tutors and students in directing and initiat- 
ing activities. It also continued to raise 
questions about improvement in reading 
abilities beyond 50 hours. The third study 
(1985-1986) v/as designed to "answer defi- 
nitively" the questions about program 
impact raised in the two previous studies 
and to draw conclusions that could direcv 
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future program planning efforts at 
LVNYC. 

In the final report, Koen et al. concluded 
that LVNYC was expending roost of its 
resources for students who were not bene- 
fiting fi-om tutoring over time and that the 
agency should m^e other options avail- 
able to students after the initial 50-hour 
period. In their recommendations to pro- 
gram directors and staffs the evaluators 
argued for concentration on the first 
instructional cy'cle, an increase in outreach 
efforts to attract more students who fit 
their kind of program, and the establish- 
ment of linkages with other literacy pro- 
grams in New York City for referrals. 
The recommendations tiius urged an 
expansion of service to new students 
rather than the development of services 
more appropriate for those who remain in 
the program over time. 

It is not clear fi'om the evaluation report 
whether these recommendations were con- 
gruent with staff perceptions of the pro- 
gram's goals. Furthermore, the stud/s 
findings depended on scores fi'om stand- 
ardized tests, which many consider of 
limited value in determining adult devel- 
opment in reading and writing. The 50- 
hour evaluation cycle may be an arbitrary 
time frame within which to evaluate indi- 
viduals' progress. The evaluation process 
and report did, however, provide adminis- 
trators and staff with some perspectives on 
their program that are not ordinarily 
available. The evaluation design, by 
focusing on instructional processes as weU 
as outcomes, made an effort to bring 
these two critical dimensions into some 
relationship. The data gathered on what 
was going on in individual and group 
sessions, furthermore, provided informa- 
tion about actual practice as compared to 
what program planners and staff members 



saw as the ideal or intended instructional 
program. 

North Ca/olina Adult 
Basic Education 

The North Carolina Adult Basic Educa- 
tion Instructional Program Evaluation 
(Fingeret 1985), funded by the Division of 
Adult and Continuing Education, North 
Carolina Department of Community Col- 
leges, aimed to provide information that 
could improve the effectiveness of the 
ABE instructional program. Part of the 
task was to determine tiie extent to whicii 
current reading research was incorporated 
into ABE programs in North C^olina. 
Understanding the "processes and dynam- 
ics" of the instructional program required 
a multisite case study model through 
which researchers could contmue to raise 
questions as they collected and analyzed 
data and developed criteria and standards 
for interpreting the data from the per- 
spectives of study participants. Using a 
distinction made by Guba and Lincoln 
(1981) and Scriven (1972), the study 
focused not on the program's merit (its 
intrinsic value) but on its worth, that is, 
the applicability of the program or prac- 
tice in a specific context. To investigate 
worth, evaluators used the perspectives of 
stakeholders as a fi-amework for analysis; 
for example, they collected data on crucial 
concepts such as "success" that stake- 
holders defined differently. The evalua- 
tors immersed themselves in the context 
and used qualitative strategies for gather- 
ing data. These included open-ended 
interviews and observation:^, as well as 
examination of written records, publica- 
tions, and numerical data. Six programs 
were selected to serve as primary sites; 
they represented a variety of rural and 
urban population areas, different sizes of 
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student enrollment, and diversit}' in the 
extent and type of iinkages. 

Findings from the study show that admin- 
istrators, instructors, and students have 
different perspectives on instruction that 
lead to conclusions about program leader- 
ship, programmatic decision making, 
recordkeeping, definitions of success, and 
the relationship of these programs to the 
state of the art. Althougih it is beyond the 
scope of this monograph to render these 
findings and Fingeret's recommendations 
in detail, describing some of these results 
helps to clarify what types of information 
can be gained from this approach and to 
suggest how these ideas may be applied to 
internal as well as external evaluations. 

The North Carolina study found program 
personnel to be highly dedicated and the 
state itself well positioned with its 
community college system to provide 
effective service delivery. Yet the study 
argues that the program's emphasis on 
movement through the schooling hierarchy 
and the acquisition of literacy skills for 
individual social mobility reflect only some 
of the appropriate gosds of adult educa- 
tion. Some adults seek reading and writ- 
ing skiUs for community development and 
enhancement of their personal lives, both 
of which need to be regarded as Viable 
alternate routes." ABE persoimel were 
found to emphasize nurturance, respon- 
siveness, and good feelings more than 
attending to successful learning and pro- 
viding leadership. Some students saw pro- 
longed emphasis on social interaction, 
however, as interfering with their learning 
process, wheref^ others appeared to 
attend classes to meet social as well as 
educational needs. 

Other findings indicated that programs 
and programmatic decisions were 
regarded by participants and personnel at 



all levels as dictated by external forces 
rather than created proactively by par- 
ticipants. Rather than considering them- 
selves as leaders, staff tended to see 
themselves as "administrators, managers 
and responders." Dependency on com- 
mercially packaged materials and meth- 
ods, Fingered argued, indicates that 
instructors are not functiom'ng as cur- 
riculum developers. Staff tended to attri- 
bute students' departure from ABE pro- 
grams before meeting thc^ir own goals to 
external, rather than program-related, 
forces. Recordkeeping and data gathering 
were regarded as forms of accountability 
rather than methods for collecting infor- 
mation to inform program development at 
the local level. 

Contrasting definitions of success among 
program constituencies provided one of 
the most interesting categories of findings. 
In the North Carolina study, success was 
viewed by staff in terms of enrollment, 
attendance, and retention (all related to 
funding) rather than in terms of student 
progress. Fingeret reported that "if 
students attend consistently, ihe" are 
considered successes regardless of skill 
acquisition progress" (p. 179). Further- 
more, students appeared to see a partici- 
patory role for themselves that instructors 
and administrators discounted. Although 
instructors "negotiated" goals with students 
to make them increasingly "appropriate," 
judgments of appropriateness were made 
by instructoi's who did not discuss ways to 
help students develop the knowledge and 
abilities to make ihest judgments for 
themselves. Because program organiza- 
tion was focused on individual mobility 
rather than on community development, 
the program remained "primarily oriented 
to schooling and to movement un a 
schooling hierarchy rather than to literacy 
in a broader sense" (p. 180). Fingeret 
cautions against limiting the evaluation of 
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ABE to indicators such as job acquisition 
or mobility, arguing that ABE be ];egarded 
as a "Ijroad investment in the ftiture" 
rather than '^simply the first step in job 
training." 

The North Carolina evaluation also 
explored in some depth the conceptions of 
reading and learning held by administra- 
tors, instructors, and students. Fingeret 
found limitations in programs not having 
a framework "within which they can ana- 
lyze instructional problems, collaborate 
with students, or generate creative 
approaches to their teaching." She attri- 
butes this to a lack of pre- and inservice 
training and of expertise in adult educa- 
tion, instruction, and curriculum, as well 
as to the imexamined assumption that 
instrjction of adults is basically similar to 
the instruction of children. 'Instructors 
tended to "identify ABE instruction with 
emotional rather than intellectual skill or 
cognitive rewards, for themselves and, in 
some cases, for their students" (p. 181). 
Fingeret found that students had internal- 
ized a phonic, subskill model of the 
process of learning to read and were not 
aware of alternative approaches to reading 
instruction or to learning. On the con- 
trary, all perspectives converged on a 
"hierarchical subskill and social mobility 
model generated by the larger literate 
society and showing little influence by the 
cultural orientation, beliefs or values of 
the student group" (p. 12). 

' Throughout the report, Fingeret empha- 
sizes the importance to programs and, by 
implication, to the field more generally, of 
maintaining the ability to question 
assumptions and beliefs underlying prac- 
tice. She also stresses the need to exam- 
ine current theory and research related to 
literacy instruction of adults and to engage 
program staff and students in ongoing 
inquiiy. One would assume, although this 



is implicit rather than explicit in the 
author's remarks, that this would require 
a significant reallocation of such 
resources as time, materials, and sta^ 
responsibilities. Rather than assuming 
that adult literacy practitioners require 
expertise from the outside, Fingeret urges 
instructors to see themselves as research- 
ers: "Instructors should see themselves as 
testing and refining the developing knowl- 
edge base and should be able to place 
their own efforts in a broader perspective" 
(p. 16). In our view, this would require 
better chaimels or networks that provide 
practitioners, often isolated, with connec- 
tions to the wider field, including better 
access to support from professional 
organizations, universities, and other 
educational centers. 

What is clear from this evaluation is that 
the issues in one state are not unique, that 
many are consistent with issues and con- 
cerns at the national level. The North 
Carolina study provides evidence that 
although instructional program evaluations 
may be designed to be relevant to a spe- 
cific context, when richly documented and 
disseminated they can contribute more 
broadly to the knowledge base for adult 
literacy education. 



Community-based Programs 

The Association for Community Based 
Education recently (1989) completed a 
longitudinal study of nine community- 
based programs across the country. 
Intended to document program impacts, 
identify exemplary models and methods, 
and provide feedback to specific 
programs, the project (directed by Greg 
Jackson) also sought to demonstrate the 
feasibility of doing this type of com- 
parative evaluation. Program administra- 
tors worked with Jackson to identify 10 



31 



learner outcomes for measuremeiit, 
including reading, writing, and math skills; 
oral language skills; reading and writing 
activities outside the program context; the 
fostering of children's intellectual and 
academic achievement; community activi- 
ties; self-esteem; and self-determination. 
Suitable instruments were found for iBve 
of these and ACBE staff developed instru- 
ments for the others. Programs selected 
four to seven of these outcomes for 
evaluation at then- site. 

Data were collected over a period of 9 
months with changes in the measured 
outcomes assessed over 3 time intervals 
within these 9 months. Considerable 
effort was made to reduce the anxiety of 
participants and to ensure that observed 
differences were caused by the program 
rather than by other factors. Althou^ all 
nine chose to have self-esteem evaluated, 
six of the nine programs selected reading 
skills to be measured and five selected 
writing skills. Other areas were evaluated 
at just a few sites where they were 
regarded as appropriate. The evaluators 
also looked at dropout and attendance 
patterns. Ninety-five percent of the 
students said they were participating in 
the program *'to be able to do more for 
myself,** and most reported being engaged 
in several forms of learning outside of 
classes or tutoring sessions. Jackson 
reports that learners were cooperative 
with the evaluation process although both 
they and their teachers found the process 
somewhat disruptive. The self-esteem and 
self-determination instruments elicited 
strong reactions, both positive and nega- 
tive. The report indicates that program 
administrators appreciated the variety of 
outcomes available for measurement and 
the assistance in the documentation of 
program impacts that could be used for 
fund-raising and *'identifying program 
weaknesses.** 



The A 3E study, which focused entirely 
on learner outcomes, provides an 
approach to program evaluation not avail- 
able in the literature before this time, 
which has imphcations for other umbrella 
organizations or agencies interested in 
multisite program evaluation designs. 
Programs were found to vary widely on 
the different measures, includiig reading 
skills, although almost all had s^iitistically 
significant impact on self-esteem. Using 
a range of instruments, it was possible to 
document some Ui. the impact of these 
community-based programs and lend 
strength to the argument tiiat they are 
effective. Because few data were col- 
lected on program processes, the study did 
not provide information about connections 
between particular features of these pro- 
grams and program impacts, that is, 
between implementation and outcomes. 
Some participating program staff were 
critical of the use of conventional mea- 
sures and expressed the wish to capture 
outcomes beyond basic skills areas, 
particularly because of the community- 
based nature of their programs. 

The ACBE report includes an extensive 
discussion of the advantages and disadvan- 
tages of different aspects of this program 
evaluation process, and it suggests that 
future evaluations refine and extend this 
design and use case studies or ethno- 
graphies in place of; or as a complement 
to, these processes. Furthermore, the 
report recommends that financial and 
technical assistance be provided to literacy 
programs so that staff can "develop and 
implement their own evaluation processes*' 
and learners can "develop and use self- 
assessment and participatoiy research 
processes** as well. These recommenda- 
tions demonstrate ACBE's responsiveness 
to the feedback of program st^if and 
understanding of program evaluation as 
an evolving, dialogic process. Further, this 



reUtionship between ACBE and 
community-based programs serves as a 
model for interactive plaiming and evalua- 
tion between umbrella organizations and 
program sites. As a multisite program 
evaluation focused on an array of learner 
outcomes, the ACBE study shows how 
data of value to program staff and learn- 
ers inside programs can be collected by 
outside evaluators. As this strategy of 
comparing programs with different empha- 
ses is revised and refined, ideas for other 
multisite evaluations will undoubtedly be 
generated. 



Technology for Literacy Project 

An example of an in-depth formative and 
summative program evaluation is provided 
by the Technology for Literacy Project 
(Turner and Stockdill 1987), a program 
that relies primarily on tlie uses of 
computers for adult literacy education. In 
this case, an evduation design for the 
project was determined after the project 
goads were decided but before they began 
to be implemented. A "formative evalua- 
tor" designed systems for data collection, 
provided onsite advice, and conducted 
workshops on evaluation as part of staff 
training. Data were collected and ana- 
lyzed during the first 2 years of the project 
for use by the project manager in enhanc- 
ing program operations. The summative 
evaluation, conducted by a national expert 
in program evaluation, Michael Quinn 
Patton, was completed in the second year 
of the project's operation and became a 
formative document used in the design of 
year 3 goals and objectives. 

The evaluation focused on direct service 
and addressed questions about recruit- 
ment, achievement, and retention. A 
second important component was training; 
the project was designed to train teachers. 



administrators, graduate students, and 
volunteers to incorporate technology into 
their programs. In addition, data were 
sought about incentive grants (the extent 
to which the center's efforts stimulated the 
use of technology by other programs) and 
research grants (the ways Uiat the center 
functioned as a laboratory for research on 
the uses of technology with adult literacy 
learners). Finally, evaluators inquired into 
the overall strengths and weaknesses of 
the program and ways to build on or 
address them. Data collection methods 
included quantitative measures such as 
standardized achievement tests, daily logs 
to record time on task, and criterion- 
referenced 'ests designed by program 
staff. Qualitative data were gathered 
through group interviews, exit interviews, 
student questionnaires, learner activity 
logs, and case studies. In addition, 
telephone interviews and follow-up letters 
were used to collect data on why learners 
left. Learners, staff; Lstructors, and 
volunteers fiUed out questionnaues twice 
a year to identify program strengths and 
weaknesses. As the evaluation evolved, a 
cost component v/as added to the evalua- 
tion design to address the interests of 
funders. Because 'project staff members 
were willing to collect data and learn 
about evaluation** (T\imer and Stockdill 
1987, p. 17), the evaluation process was 
fully integrated into the project 

The project report (written by Michael 
Quinn Patton) includes a fiill description 
of the smnmative external evaluation 
intended to assess independently the 
effectiveness of project implementation 
and outcomes. The report describes the 
implementation of each component of the 
Technology for Liteiacy Center and 
reports nutcome gains in literacy skills and 
attitudes. Tht evaluation report empha- 
sizes that no single indicator is sufficient 
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to capture the impact of the project on 
learners, but rather that there is a need to 
create an "outcomes mosaic or tapestry" 
and then to stand back and view the 
picture of program impact as a whole. 
That there is "no average TLC student" is 
reported as a "very dramatic, important 
and significant finding." IXX^ is an indi- 
vidualized, self-paced program serving a 
very diverse population of learners. Eval- 
uators stress the importance of not placing 
too much emphasis on average achieve- 
ment scores or average hours of participa- 
tion and of avoiding the use of these 
averages to influence learners' expecta- 
tions for their own progress. The report 
includes sensitive examination and discus- 
sion of issues related to attrition, progress, 
and completion. Attrition is related to 
the many obstacles, minor and major, 
encountered by adult learners and the 
resultmg "irregular path of learning." 
Computing dropout and completion rates 
poses major definitional FiOblems, the 
report argues. This finr^lng raises ques- 
tions about the extent to which programs 
can attribute attrition to factors inside or 
outside the program^ 

An extensive section of the summative 
evaluation report details attitudes toward 
the program of learners, volunteers, and 
professional staff. This information was 
collected through focus group interviews, 
analysis of questionnaires, and two in- 
depth qualitative case studies of learners. 
In the focus groups, learners were asked 
how they learned about the program, their 
first impressions, their reactions to using 
computers, what they Uked and disliked 
about the program (for example, its loca- 
tion in a shopping mall), and its personal 
impact on them in learning to read, as 
well as information about their motivation 
to learn. The interviews with volunteers 
addressed questions about their traimrg, 
uncovering theu- felt need for more than 



the standard materials-based training and 
more experience with computers. Profes- 
sional staff were found to share common 
values and to have a strong sense of inno- 
vation. In addition, they exhibited an 
unusually strong commitment to helping 
learners document their Laming and to 
keeping good records for program man- 
agement and accountability. Findings 
related to the technology training com- 
ponent argue that despite its additional 
demands, these experiences contributed 
not only to helping others, but to making 
the center staff more reflective and 
effective. 

An important feature of the design was 
the clear distinction made between forma- 
tive and summative evaluation. Patton 
regards the project as a "textbook exam- 
ple" of "how to conduct evaluation in 
accordance with the standards for educa- 
tional evaluation, namely, that the evalua- 
tion be useful, practical, ethical, and 
accurate" (Turner and Stockdill 1987, 
p. 24-6). Although the research compon- 
ent of the project has not yet been rea- 
lized, its training and evaluation efforts 
seem to have enhanced direct services. 
According to Patton, the program has 
created a "culture of learning" that applies 
to both staff and students. Patton points 
to the positive oiiganizational culture that 
makes the program "not a place for 
schooling, but a place for learning." As an 
example of a comprehensive and sophisti- 
cated program evaluation process, the 
Technology for Literat^r Project ev iluation 
is useful for projects with both similar and 
different goals and orientations. The 
thoroughness of the report provides much 
material for discussion about the relation- 
ship among program philosophy, evalua- 
tion methods, and outcomes. 
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Broad-based Systemic Approaches 

The current literature on program evalua- 
tion in adult literacy includes a few 
examples of systems designed to connect 
related programs through information 
gathering, analysis, and dissemination. As 
resources for others who may be planning 
program evaluation, they illustrate differ- 
ent strategies for meeting the needs of 
diverse stakeholders and for networking 
programs with each other. 

Many dty and state systems undoubtedly 
have or are in the process of developing 
databases for adult literacy programs in 
different geographical areas. However, 
the Adult Literacy Information and Eval- 
uation System (ALEES) in New York Qty 
is probably the largest and perhaps most 
ophisticateJ microcomputer system cur- 
rently in use in the adult literacy field. 
Drawing on data from the four literacy- 
providmg agencies in the city (the Board 
of Education, Gty University of New 
Ycrk, Community Development Agency, 
and the public libraries), AT.TRS is a 
uniform, citywide database that contains 
both program-related and individual 
student data coded to maintain adult 
learner confidentiality (Cook, Denny, and 
Weickert 1985; Schneider, Cook, and 
Schwarz 1988). 

The ALIES system is designed to have 
four interrelated functions, although only 
two are actively in use at present The 
first is tracking-collecting descriptive 
information about program implementa- 
tion through numbers and demographics 
related to students and numbers of ^'ter- 
acy programs and services offered (includ- 
ing contact and instructional hours). The 
second function is monitoring, or collect- 
ing information to determine the degree 
to which programs are meeting goals spe- 
cified in their contracts (the discrepancies 



between the original program design and 
the actual program operation). These 
data are intended to be used by program 
managers and decision makers for 
monitoring individual programs, although 
their usefuhess at the program level has 
:iot been studied. 

The other two functions, as d( scribed in 
the original design for the system, are just 
beginning to be implemented. In the orig- 
inal design document, the database was 
also intended to be used for research and 
evaluatioa It is unclear from the original 
document, however, at what level of the 
system the questions for this research 
would originate, what types of data would 
need to be gathered, or what role individ- 
ual programs would play. The fourth 
function of AUES, '*in-depth investiga- 
tions'* of the adult literacy instructional 
process and the effects of instruction on 
individual participants and their families, 
is still to come. 

To date, researchers at the literacy Assis- 
tance Center working with external eval- 
uation consultants have collected a wide 
range of demographic and outcome data 
on students in basic education and ESOL 
programs to study the simultaneous 
impact of several variables related to 
program effects. Although data on a wide 
range of independent variables are being 
collected, dependent variables are cur- 
rently limited to scores on the standard- 
ized tests (such as the Tests of Adult 
Basic Education and the John Test for 
English for Speakers of Other Languages). 
There is no currently available documen- 
tation of whether these data are actually 
being used at the program level. Reasons 
for this may include the difficulties in 
synthesizing and reporting information 
from such a large database in a timely 
fashion and the degree to which programs 
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regard test scores as useful in planning 
and development. 

In contrast to the ALIES system, which 
has been designed primarily for citywide 
monitoring, a comprehensive and innova- 
tive system designed for interactive pro- 
gram evaluation at local and state levels 
has been developed by the Massachusetts 
Worlqjflace Education Project (MWEP), 
which included 16 sites across the state as 
of -^ril 1988. The goal of the project is 
to strengthen the state economy by 
upgrading the skills of the work force and 
giving workers access to classes provided 
at the workplace that will enhance theu- 
opportunities for job advancement (Hikes 
1988b). Workplace-speciilc curricula have 
been developed by all projects, although it 
not clear whether this was done by lit- 
eracy educators alone or by educators in 
conjunction with workers. 

The MWEP is distinctive in that its pro- 
cess for program evaluation not only 
provides extensive data about individusd 
program functioning but links this infor- 
mation to the concerns of policy makers 
at the state level. An outside evaluation 
team of consultants has encouraged 
formative rather than summative evalua- 
tion by developing a guidebook describing ■ 
critical features of successful workplace 
education projects and a framework for 
project coordinators' use in documenting 
the progress of their projects. Further, by 
interviewing workers about their participa- 
tion in workplace classes, the team was 
able to develop educational guidelines for 
workplace educational projects. Recently, 
the research team has interviewed man- 
agers, supervisors, and imion leaders, as 
well as workers from the previous sample, 
about the impact of the basic education 
courses on theu- worlq>lace. As in other 
program evaluation results cited previ- 
ously, increased self-esteem was the most 



significant outcome reported by workers 
of all ages. The most successful curricula, 
the evaluations found, responded not only 
to the work requirements of participants 
but also to their broader educational and 
life needs. Although difficult to quantify, 
the increase in the amoimt and quality of 
communication among people in the 
workplace has been significant. 

The MWEP evaluation process has sev- 
eral other unique features. Programs 
routinely collect student assessment data 
using locally determine d methods. When 
each project begins, it sets up student and 
program goals and reports progress to 
funders in a standard quarterly report 
format developed by the program eviua- 
tors. A central feature of the ongoing 
evaluation process is a representative 
advisoiy board established at each pro- 
gram composed of management, union 
representatives and/or workers, the pro- 
ject coordinator, and a representative 
from the local service delivery office. The 
board meets regularly to discuss the pro- 
ject, including workers' progress and the 
curriculum, and to suggest changes in 
project direction. Program evaluation is 
an ongoing process that reflects the 
diverse viewpoints and unique character- 
istics of each workplace context. 

Because each workplace education project 
integrates data collection systematically 
with program functioning, a cumulative 
picture can be created that contributes to 
planning the direction of the workplace 
education project as a whole. The pro- 
ject's evaluation design includes a man- 
agement information system with data 
entered monthly at local sites, so that they 
can be retrieved at the central office in 
Boston by members of the steering com- 
mittee or the program coordinator. Al- 
though the leadership acknowledges the 
need for further development in specific 
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areas (such as learner assessment), the 
overall strategy for evaluation has success- 
fi*lly linked local program evaluation func- 
tions with over^ statewide evaluation. 
Evaluation processes at all levels have 
reflected ii^ut from a broad range of 
stakeholders, including funders, teachers, 
union leaders, and company managers. 
The creation of a statewide profile do<>s 
not constrain curriculum and assessment 
spccinc to the setting and population 
served. Furthermore, because iht evalua- 
tion process involves collaboration among 
diverse stakeholders at both local and 
state levels, it strengthens relationships 
among people with different roles in the 
workplace and enables individual pro- 
grams to be self-critical without experi- 
encing a threat to their continuation. 



Conclusion 

The literature of program evaluation 
points to several interesting issues related 
to evaluating program goals, practices, and 
outcomes. It suggests the importance of 
calling into question the program goals as 
an essential part of the evaluation process. 
Rather than seeing these goals as fixed 
and unchangeable, program staff^ If arners, 
and others involved in the evaluation can 
regard the goals or mission of the pro- 
gram as subject to scrutiny and change. 
Rich data on program practices are essen- 
tial for connecting outcome or impact 
measures with implementation. Even if 
an array of measures of learner progress 
is used, it is difficult to interpret the 
findings when little is known about the 
processes of teaching and learning within 
the program. Expanding the outcome 
measures for learner progress is also 
needed. Some of the studies cited in this 
chapter demonstrate that focusing solely 
on gains in reading and/or writing fails to 
capture unintended outcomes or oUier 



significant changes, such as learners* 
enhanced self-coucept, increased commim- 
ication among workers and managers in 
the workplace, community participation, 
reading or writing outside the program, 
meeting one's own objectives (such sif 
getting a driver's license), a£fecting the 
intellectual lives of children, returning to 
the program as a staff member or volun- 
teer, or participating with others in effect- 
ing social change. Furthermore, quite dis- 
parate programs can be compared if they 
select from a common list of alternatives 
those measures that best suit their own 
goals and purposes. Finally, rther pro- 
gram outcomes, such as patterns of attri- 
tion and rett^iiuon or stsif and volunteer 
attitudes, n&cd to be examined. The few 
examples of program evaluation that 
include thoughtful analyses and interpreta- 
tion of such program data provide models 
and directions for future work in the field. 

The literature reviewed lacks examples of 
program self-evaluation that would clearly 
add to an understanding of the roles that 
staff, managers, and learners nan play in 
an evaluation process even without the 
direct assistance of outsiders. The 
external evaluations show how evaluators 
can play a variety of roles including selec- 
tion and design of learner outcome mea- 
sures, overall design of formative and 
summative plans for evaluation, as well as 
data collection, analysis, and interpreta- 
tion. In several of the evaluations 
described, the process has been more or 
less collaborative with program staff and 
learners. These collaborations have 
served a number of purposes, among them 
informing programs about conducting 
evaluation processes an j bringing to the 
evaluation itself the perspectives of 
different stakeholders. Another point 
concerns the benefits and drawbacks of 
qualitative evaluation methods. Although 
there is considerable evidence that 
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qualitative data contribute significantly to 
evaluation, this type of infonnation is 
harder to collect and analyze. Programs 
may require some assistance in gathering 
and managing qualitative data. Finally, 
several of the recent program evaluations 
document some of tbe stress ant 
dissonance created by the evaluation pro- 
cess (ACBE 1989; Tamer and Stockdill 
1987). Including data in evaluation 
reports about the affective responses of 



participants to program evaluations is 
educational for others. Evaluation is a 
learning process, like other forms of 
learning, it is occasionally uncomfortable 
and involves risk taking and trust. By 
describing some of the reactions of teach- 
ers and learners to particular aspects of 
the evaluation, these reports provide a 
realistic picture of the affective dimen- 
sions of evaluation (For a summary of 
approaches to program evaluation, see 
table 1.) 
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TABLE 1 

APPROACHES TO PROGRAM EVALUATION 



TYPES o Single site/multisite 

o Formative/ summative 

o Implementation/impact 

PURPOSES o Plan 

o Secure and maintain funding 

o Provide staff development 

o Improve instruction 

o Allocate resources 

o Assess research utilization 

o Compare perspectives of stakeholders 



LEVELS 



EMPHASES 



METHODS 



o Single program 

o Citywide system 

o State system 

o National 



o Program mission and goals 

o Learner experiences and outcomes 

o Instruction 

o Staff development 

0 Culture of program as workplace 

o Enrollment/recruitment/retention 

o Grouping practices 

o Written curriculum 

o Materials utilization 

o Community relations and outreach 

o Governance and management 

o Financial resources and expenditures 

o Support services 

o Refenal systems 

o Recordkeeping 

o Observation 

o Interviews (group and individual) 

o Site visits 

o Surveys and questionnaires 

o Document/statistical analysis 

o Logs and journals 

o Audio and video recordings 

o Case studies 
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LEARNER ASSESSMENT 



Introduction 

Through a review of the literature, this 
chapter describes and evaluates four 
major approaches to learner assessment: 
standardized testing, materials-based 
assessment, competency-based assessment, 
and participatory assessment. Its purpose 
is to assist adult literacy educators and 
policy makers in understanding assessment 
procedures currently in use and in making 
informed choices about selecting or con- 
structing procedures for learner 
assessment 

Although its purposes, methods, and 
instruments vary, learner assessment can 
broadly be defined as a process of collect- 
ing and analyzing data provided by learn- 
ers in order to make judgments about the 
literacy accomplishments of individuals or 
groups. Learner assessment comes in dif- 
ferent forms, at different points in a 
learner's stay at a program; it may empha- 
size different views of literacy and learn- 
ing and yield distinct types of information 
to diierent stakeholders. What it hopes 
to accomplish is wide-ranging and com- 
- plex: the provision of information useful 
to teachers in planning instruction, to 
learners in determining &eu own progress 
towards particular go^, to program man- 
agers and staff in evaluating their instruc- 
tional impact on learners' lives, and to 
funders in ascertaining a degre* of pro- 
gram accountability and success. 



Although learner assessment is clearly not 
the only element of pro[ "am evaluation, it 
is nonetheless a key ture of it, of 
interest to program stuii, learners, and 
funders as an indicator of a program's 
success. Yet, most programs have at best 
only limited options in selecting assess- 
ments. Choice is made difficult by a 
number of factors: the needs of different 
audiences for different types of informa- 
tion, requirements by many funders for 
particular measures, the limited number 
of options readily available, and the lack 
of accessible information about the 
strengths and weaknesses of different 
assessment approaches. Too often both 
the goals and processes of assessment are 
narrowly conceived. These limitations 
frequently emanate from particular con- 
cepts of literacy, learning, and teaching. 
As a result, entry and exit data, as well as 
ongoing assessments, may not provide 
enough direction to program staff and 
learners about improving the quality of 
the program. This is a critical problem 
because the processes and products of 
learner assessment are central to the 
quality of data collected for program 
evaluation, ^ 

This chapter draws upon a literature base 
that includes descriptions and samples of 
assessment processes and products both 
commerci' My produced and locally devel- 
oped, as well as examples of assessment 
research, surveys of research literature 
related to reading measurement, and 
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published reviews of assessment 
approaches. 

The review of these four approaches 
addresses their philosophical orientation 
and distinguishing characteristics, their 
strengths and imitations, and the criteria 
suggested in the literature for selecting or 
constructing procedures for learner assess- 
ment. Implicit in this discussion is a 
consideration of assessment and decision 
making: the degree of choice and control 
that program managers, staff, and learners 
can exercise within particular assessment 
approaches. 

This section is divided into four parts, 
each covering a major assessment 
approach. (See table 2 for a list of 
features of each type.) The emphasis 
here on formal assessment should not be 
interpreted as a devaluing of the ongomg 
informal assessments that many adult lit- 
eracy instructors, often with learners, 
conduct as part of the learning process. 
Instructors assess all the time. Although 
some may keep teaching logs in which 
they describe activities and results, all rely 
to some extent on the informal observa- 
tions of learning that are part of the 
everyday life of classes and tutoring 
session;^. Many of the activities associated 
with informal assessments reappear as 
features of the more formal approaches 
described here. These activities include 
reading inventories, learning profiles, and 
writing portfolios, which rely on data pro- 
- vided by learners themselves about tiieir 
progress over time. Assessments of pro- 
gress are also conducted through confer- 
ences with learners in which instructor 
and learner both review the work done to 
date. In other conferences with instruc- 
tors, learners might read aloud and then 
retell what has been read, thereby provid- 
ing data for a type of miscue analysis of 
their reading. The primary audiences for 



these informal assessments are the instruc- 
tor and learners. By providing informa- 
tion to tlie instructor about each learner 
and to learners themselves about what has 
been accomplished, these informal assess- 
ments play a critical role m motivating 
and involving learners, as well as in 
curriculum planning. Soiue reading 
researchers have called for the synthesiz- 
ing and reporting of information about 
informal assessment procedures and data 
(Farr and Carey 1986), Such information 
would be useful for teachers and teacher 
educators, as well as for others involved in 
literacy assessment 



Standardized Testing 

Many of the standardized tests of reading 
used in adult literacy programs are norm 
referenced-they measure a learner's per- 
formance relative to the performance of 
others who have taken tibe test. Even 
when "functional" real-world texts are 
used, the tasks for the reader are skill 
based. Most of these tests measure tradi- 
tional reading components such as vocab- 
ulary and comprehension, as well as spell- 
ing and arithmetic, and they typically yield 
grade equivalent scores. la contrast, the 
ACBE (1989) program evaluation 
describes a standardized test of self- 
esteem used by some community-based 
organizations. Thite of the standardized 
tests most widely used in adult literacy 
programs are the Adult Basic Learning 
Examination (ABLE), the Tests of Adult 
Basic Education (TABE), and the Wide- 
Range Achievement Test (WRAT). (See 
Jackson 1990 for a comprehensive review 
of tests for adult literacy programs.) 
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TABLE 2 

APPROACHES TO LEARNER ASSESSMENT 



TYPE 



FEATURES 



I. Standardized Testing 



II. Materials-based 



ni. Competency-based 



IV. Participatory 



o Norm-referenced 

0 Literacy as reading skills 

0 Grade equivalent scores 

o Administrative usability' 

o Cost-effective 

0 Independent of curriculum 

o Commercially available 

G Related to progress in predeter- 
mined materials 

o Literacy as reading skills 

0 Systems management model 

0 Specific real-life tasks 

0 Predetermined standards of 
performance 

0 Competence in defined areas 

o Continuum of difficulty 

0 Administered fi-equcntly for feed- 
back and advancement 

0 Broad range of strategies possible 

o Processes and products 

0 Literacy as practices ani critical 

reflection 

0 Range of texts, tasks, and contexts 

o Active role of learners 

0 Co-investigation 

0 A form of teaching 

o Ongoing 

o Broad range of strategies possible 
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There are a number of reasons why stand- 
ardized tesLs are the most widely used 
approach to learner assessment in adult 
literacy programs in the United States. 
l\vo features that make them attractive 
are their administrative usability (Nagger 
ct al. 1975) and cost-effectiveness. Stand- 
ardized tests are relatively easy and 
inexpensive to administer; large groups of 
learners can take them at the same time 
under the supervision of few program 
staff. Further, relatively little training is 
needed to enable staff to administer the 
test 

The information gathered from standard- 
ized tests provides ostensibly objective 
information about gains over time. Easily 
aggregated and reported, standardized test 
scores are viewed as an attractive index of 
program functioning by funders, legisla- 
tors, and government agencies. By con- 
necting test scores to computerized man- 
agement information systems (Schneider, 
Cook, and Schwarz 1988; Taggart 1986), 
managers can establish correlations 
between test scores and other program 
variables such as attendance, and 
umbrella organizations or funders can get 
a view of individual program impact 
relative to an entire initiative. 

Standardized tests are useful in providing 
an indication of how someone reads, but 
the relation of such indications to actual 
reading behaviors must be infened (Farr 
and Carey 1986). They provide evidence 
of how well someone reads from a limited 
perspective, under a limited set of condi- 
tions, and with a limited set of responses 
(ibid.). As one of a number of indicators 
that together provide a more complete 
picture of student accomplishment, test 
scores can contribute to the information 
needed to estimate learners' reading 
development and evaluate a program's 
success in achieving stated goals. 



Norm-referenced standardized tests have 
been frequently jnisused to provide mea- 
sures of individuals' reading ability and 
inform instructional decisions for individ- 
uals. Some ar .e (Chall 1990) that these 
tests are useful within the purposes they 
were originally designed to serve, that is, 
they are best used to obtain survey infor- 
mation, to sample groups of adults in 
order to determine general degrees to 
which people might be said to be literate. 

In some cases, standardized tests function 
as fairly accurate predictors of success. 
Although a low score on a standardized 
test is not a good predictor of an individ- 
u^ learner's ability to do well cn other 
types of reading tasks, a high score is a 
good indication that a learner will be able 
to handle different reading materials in 
various contexts (Sticht 1990). Further, 
for leamers whose goal is a GED certifi- 
cate or entry into an employment training 
program requiring a minimum test score, 
these tests reveal how close leamers are 
to the accepted measure for entering 
these programs. As long as test scores 
remain the criteria for entry, the ability to 
achieve on tests assumes great 
importance. 

In spite of their appeal, standardized tests 
have been subjected to a rigorous critique 
by both researchers and practitioners in 
adult literacy education. An in-depth 
re''^ew of these concerns is beyond the 
scope of this iQonograph; however, several 
key issues arel-elevant here. 

The most obvious problem with standard- 
ized tests is their reliance on grade-level 
equivalents. When an adult is identified 
as reading at a 32 level, this denies the 
relevance of the abundant life experience 
and knowledge that adults bring to the act 
of learning and employ in their interpre- 
tations of texts and setting of learning 

44 



ERIC 



51 



goals. An adult, reader is not equivalent, 
therefore, to a third-grade reader. Hie 
converse is also problematic; the ability of 
adults to score 5.0 on a reading test pro- 
vides no assurance that they would be as 
proficient as fifth graders in other literacy 
tasks (Donlon cited in Buros 1978; Sticht 
1990). Further, adult learners invest these 
scores with disproportionate significance 
so that progress defined by oUier mea- 
sures is devalued. This is particularly 
common in those programs where the 
emphasis is on testing out of the program 
(for example, to seek a GED ceitificate). 

Another difficulty in the use of standard- 
ized tests involves the relation of the tests 
to a program's instructional model. The 
belief that identify reading levels 
leads directly to the placement of adults 
according to graded reading materials. 
Teaching and learning are consequently 
reduced to the management of adult 
learners' progress through predetermined 
sequences of these materials. Because 
adults are no longer in the K-12 school 
system and their needs for learning are 
very different firom those of children, 
standardized grade-school normed tests 
are not useful for matching adults to 
instructional programs (Sticht 1990). 
Further, standardized tests are not 
designed to reveal what learners have 
accomplished through a program's instruc- 
tion. In fact, there is often a wide gulf 
between the instruction provided in a 
program and what is assessed in a stand- 
- ardizedtest. Many programs and teachers 
attempt to respond to learners' goals and 
intentions by negotiating curricula tied to 
the ways in which adults use and wish to 
use literacy in their lives. Yet, in an 
effort to respond to the demands of the 
test, some literacy educators still feel 
compelled to provide practice exercises 
similar to what learners are likely to see 
on ihe TABE or the ABLE. Sometimes 



it is a struggle for learners to see the 
relationship between a multifaceted 
curriculum und progress as determined by 
a standardized test 

Even if these tests make sense within a 
program's curriculum, the process by 
which they are given raises questions 
about the validity of their results. In 
some cases, because of instructors' desire 
to minimize the negative effects of test- 
taking on adult learners, the conditions of 
test administration become less secure. 
They are administered under a variety of 
circumstances, sometimes with assistance 
provided to the test-taker. Tlie same 
forms of the test are given many times to 
the same learners and are often available 
to them in the regular ftmctioning of the 
progranL Scores on standardized tests 
that appear objective may reflect radically 
different conditions of administration. 

Critics of these tests raise further ques- 
tions about their appropriateness for adult 
learners. Some have challenged the 
assumptions underlying Ihe tests, for 
example, that a useful portrait of an 
adult's ability at reading can be ascer- 
tained by responses to passages read 
outside of a meanii^l context 
(Hieronymus cited in Buros 1972; Hill and 
Parry 1988). Further, for msiiy adults, 
standaidized tests are closely associated 
with past school failure and therefore 
cauiie considerable anxiety by reinforcing 
a deficit model of performance in reading. 

Still further, the format of standardized 
tests as collectioas of passages with 
questions that have single right answers 
demonstrates a viev/ of reading that 
denies the possibility of multiple readings 
of texts or of texts read for a v^ety of 
purposes. Indeed, the act of taking a test 
requires skills beyond those being tested: 
learners must be familiar with test 
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structure and tasks in order to perform 
well. Sticht (1990) points out that an 
assumption underlying standardized tests 
is th9t in reading, skills precede knowl- 
edge. By assuming that it is possible to 
design tests in which prior knowledge is 
inelevant to answering questions, test 
makers create an aura of objectivity but 
sacrifice validity. Finally, though few 
would argue for the direct evaluation of 
writing by standardized tests, the omission 
of writing suggests an implicit definition of 
literacy as decontextualized reading skills 
(Fan and Carey 1986). 

Researchers and test makers are seeking 
to improve standardized tests. The use of 
cloze passages in the Degrees of Reading 
Power (DRP), for example, indicates a 
view of reading unlike that of the norm- 
referenced tests described earlier. The 
DRP reflects more current views of the 
reading process as the construction of 
meaning, although the passages to be read 
are still out of the context of individual 
readers* lives. A newer version of the 
TABE (1987) endeavors to reflect lan- 
guage and content appropriate for adults 
and to measure the understanding and 
application of conventions and principles. 
However, Hill and Pany (1988) argue that 
by including readings meant to provide 
higher interest to adult learners and/or to 
present fiinctional situations, the writers of 
the new TABE confuse the test takers 
even further. Although the material to be 
read is appropriate to adults and of high 
interest, the items still assume that 
meaning is to be derived from the text, 
not from the knowledge and experience of 
the reader. Thus, it is difficult for the 
reader to distinguish whether literal or 
interpretive responses are called for (Hill 
and Parry 1988). 

Adapting Street^s (1985) theoretical dis- 
tinctions. Hill and Pany (1988) suggest a 



shift in tests like the TABE from an 
"autonomous" to an "ideological" or 
"pragmatic" model of literacy. They 
suggest that the reading comprehension 
component of the test be divided into two 
sections. Tbe section containing 
autonomously oriented material would 
concentrate on reading skills and practices 
thought to be useful and stable among 
most readers. This section could include 
some of the origmal TABE material such 
as tables of contents and encyclopedic 
materisd, along with material that 
demonstrates "how things work and how 
to do things" (HiU and Pany 1988, p. 56)- 
manuals and pamphlets that have fimc- 
tional importancs for people but that up 
to now have been up absent from the 
TABE. The pragmatic section would 
include the kind of material-advenising, 
letters, poetiy-characteristic of the new 
TABE (or be further divided into use of 
a reader's imaginative powers and com- 
mon sense reasoning). In this section, the 
meaning would be assumed to reside not 
in the text but in the context that readers 
construct from it (Hill and Pany 1988, 
p. 57). 

Sticht (1990) speculates that standardized 
tests may be improved technically through 
the application of item response theory 
(IRT), a psychometric theoiy that has 
already been applied to a number of tests 
including the NAEP study of literacy in 
young adults (Kirsch and Jungeblut 1986). 
IRT identifies test items and attributes to 
them specific characteristics. In L T, 
certain factors such as item difficulty r.;e 
taken into account, ^^^en considered c u^- 
ing test development, IRT enables test 
constructors to have a more complete 
understandmg of the factors that comprise 
test performance. IRT can also be 
applied to scoring and analyzing test 
results, producing a fairer and more 
accurate scale for each learner (Farr and 
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Carey 1986). Using IRT, NAEP is cur- 
rently adapting its tests of prose, docai- 
ment, and quantitative literacy for wider 
use in developing literacy profiles of : iult 
learners (Kirsch 1990). Critics of IRT 
assert, however, that separating items for 
analysis from their context is not appro- 
priate. The order of test items affects 
response and, in the case of cloze items, is 
dependent upon context (Johnston 1984). 



Materials-based Assessment 

In materials-based assessment, learners 
are evaluated on the basis of tests or 
"check-ups" taken upon completion of a 
particular set of materials. like standard- 
ized tests, this form of assessment is 
readily available through commercial pub- 
lishers who advertise the ease of adminis- 
tration ard systematic approach to teach- 
ing and learning. These assessments are 
meant to be used separately or in conjunc- 
tion with standardized testmg. 

Probably the most popular and wide- 
spread example of (he materials-based 
instruction/assessment approach is 
produced by Laubach Literacy Action 
(LLA), whose volunteers serve more than 
700 conununities in 45 states (Gadsden 
1988). The materials are designed 
exclusively for individualized tutoring, 
although they have been used by other 
programs more eclectically. Materials 
used to tutor adults include self-paced and 
sequentially presented eveiyday literacy 
tasks (for example, writing checks, filling 
out applications), short readmgs, and 
phonics exercises (Rice 1983). These 
materials rely on a discrete skills approach 
to teach basic reading, writing, and listen- 
ing skills at 0-5th.grade reading ability. 

For the purposes of assessment, learners 
complete skill book check-ups 



aiministered, scored, and recorded by the 
tutor, who then reviews the results with 
learners at the following session. The 
objectives of the evaluation are measuring 
student progress in relation to the skill 
books, diagnosing strengUis and weak- 
nesses, and building test-taking confi- 
dence. Materials-based assessment pro- 
vides the instructor and learner with 
information about how well the learner 
performed on specified tasks related to 
the curriculum. Althou^^ the materials 
stress that the check-ups are only one way 
to measure progress (Laubach, Krik, and 
Laubach 1981), no systematic assessment 
procedure is offered to document learners' 
performance or their uses of reading and 
writing outside the tutoring sessions. 

Materials-based assessment is similar to 
basal reader tests for elementary school 
children that are administered at the end 
of each book and/or at the end of each 
unit in the book. Critics of basals main- 
tain that the majority of postreading ques- 
tions typically focus on rccaU of detail 
rather than understanding of characters' 
motivations and feelings; such questions 
do not engage readers in comprehending 
and appreciating what they have read. In 
some material there seems to be an 
assumption that less competent reade rs 
should be asked only the recall questions 
(Weaver 1988). These concerns apply as 
well to materials-based assessment for 
adults. 

The materials-based approach offers 
another ex^?mple of the literacy as skills" 
perspective nested in a broader "systems 
management model" (Fingeret 1984, 
p. 31). This skills perspective is also 
apparent in the view of writing promul- 
gated in the materials. Writing is defined 
as transcription and assessed as spelling; 
no allowance has been made for Icamers 
to compose their own texts, and no way of 




examinmg those texts over tiroc has been 
suggested. 

The materials-based approach to assess- 
ment makes possible a close connection 
between curriculum and assessment 
However, it creates a dosed system that 
does not invite critical analysis of teaching 
processes and materials. In a materials- 
based approach, then, both the teaching 
materials and the assessment are stand- 
ardized; the materials are sequenced and 
specified, and the assessment questions 
associated with the materials are the same 
for all learners. The notion that assess- 
ment should be closely related to the 
experiences of learning is a compelling 
one, but in such a system there is little 
provision for learners to direct their own 
study, particularly when almost all the 
curriculum is prepackaged or when assess- 
ment is limited to the part of the curricu- 
lum that is. The literacy activities beyond 
the system go unassessed and are perhaps 
not recognized as meaningful by learners 
or teachers. 



Competency-based Assessment 

Competency-based literacy assessment 
measures an adult's performance on spe- 
cific real'lue tasks (such as reading maps 
or bills) against a predetermined standard 
of acceptable performance (Na&iger et al. 
1975, p. 37). Growth or progress is 
defined as the achievement of competence 
in defiLto areas. This approach to assess- 
ment is related to criterion-referenced 
tests that focus on how weU a learner 
achieves on particular test objectives 
rather than how well a learner performs 
in relation to others. This movement to 
make assessment task oriented and rele- 
vant to learners' everyday activities began 
as an alternative to the more geneial stan- 
dardized tests, which had come imder 



criticism for being unable to address adult 
literacy achievement within a more 
functional framework (ibid.). The findings 
of the Adult Performance Level Project 
have been influential m the development 
of competency-based instructional 
programs and further competency-based 
assessments (Hunter and Harman 1979). 

Competency-based adult education 
(CBAE) and asse.' ment are distinguished 
from other instructional and assessment 
models by a number of features. In 
CB^ ends or outcomes are always 
specified, and teaching is structured for 
explicit mastery of those outcomes. 
CBAE typically emphasizes the practical 
application of basic skills; to reflect this, 
assessment techniques vary with compe- 
tencies being achieved. Proponents cite 
its responsiveness to the needs of adult 
learners. CBAE does not have time limi- 
tations nor does it require that learners 
come regularly to programs. It recognizes 
the importance of prior leammg (Parker 
1984, p. 108) and rewards what individuals 
can already do. In some cases, competen- 
cies to be achieved are decided upon 
collaboratively between learner and 
instructor. 

Among the assessment strategies used in 
CBAE are ] ^per and pencil tests, life 
experience simulations, performances, 
portfolios, and basic sidlls tests. Some 
competency-based assessment strategies 
(hands-on demonstrations of proficiency in 
vocational skills) are designed and criteria 
for evaluation set with the input of experts 
in the areas of competence being 
assessed; this ensures a connecuon 
between demonstration of that 
competency and how it is typically applied 
in a real context Assessmei : is frequent, 
so that learners get continuous feedback 
and know fau-ly quickiy when to move on, 
an important feature in light of the time 
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constraints adult learners often have 
(Parker 1984). 

One competency program, the Central 
New York State External High School 
Diploma (EHSD), illustrates this diversity 
in assessment Although not a program 
deaUng specifically with initial literacy 
education, ^e EHSD is an innovative 
approach to granting adults their high 
school diplomas according to their accom- 
plishments in basic skills, life skills 
(including consumer, scientific, citizenship 
and health awareness, and occupational 
preparedness), advanced occupational/ 
vocational, academic skills, and specialized 
skills (community organizmg, art, music). 
Its "open testing technique" is character- 
ized by flexibility in time and location 
(including take-home tests), the use of 
several communication modes (including 
oral interviews), performance demonstra- 
tions and simulated /ife experiences, 
explicit understanding and open discussion 
with participants of competencies to be 
demonstrated, continuous feedback on 
progress, and involvement of experts in 
the assessments (Nickse 1975, p. 123). 
This approach reflects one fundamental 
assumption of the program: that assess- 
ment of competencies is most valid when 
the test environment is as close to real 
contexts as possible (Alamprese 1985, 
p. 46). 

Another example of competency-based 
assessment, the Comprehensive Adult 
Student Assessment System (CASAS), has 
an approach different from the EHSD. 
Developed under the auspices of the San 
Diego Community College Foundation 
and the CASAS Consortium (a group of 
45 agencies providing ABH, ESL, and 
vocational ESL in the state of California), 
CASAS measures reading and math 
through life skills activities along a 
continuum of difficulty (CASAS 1987a,b; 



Rickard 1988). Its master list of 
competencies was developed through an 
assessment of California learners; 
individual communities using CASAS 
select competencies from the list that 
reflect the goals and concerns of their 
s]^ecific locale or group of learners. 

Each item on a CASAS test relates 
directly to a functional life skill 
competency statement To ensure a close 
relationship between these test items and 
tasks people accomplish in life settings, 
items are field-tested as reading tasks in 
the contexts in which they would normally 
be found. Items designed to assess writing 
competency are not yet generally avail- 
able. CASAS test itenous have been devel- 
oped and standardized for difficulty using 
Item Response Theory (IRT); it is there- 
fore possible for CASAS tests to be 
customized to the competencies identified 
for particular populations. 

The CASAS calibrated test item bank 
includes approximately 4,000 test iiems. 
The tests themselves are administered 
either in paper and pencil multiple-choice 
format or as responses to oral prompts. 
At the beginning levels, these test items 
concentrate on basic tasks needed to 
function in the community or in employ- 
ment (CASAS 1987a,b), such as reading 
directions on a fire alarm box or inter- 
preting traMc signs; at the higher levels 
they seek to measure basic reading from 
passages in contexts relevant to adults and 
their goals; and at mort complex levels 
the items seek to measure criticial thinking 
through the ways in which adults fill out 
complex forms or read matrices and maps. 
CASAS tests are standardized and yield 
norm-referenced scale scores, although 
criterion-referenced scores can be made 
available. The CASAS scale is divided 
into four levels that reflect a movement ' 
from difficulty with basic literacy and 
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computational skills to the ability to 
function at a high school/GED level. The 
system attempts to report learner 
accomplishments in ways that satisfy 
fimders and yet still respond to learners' 
goals (Fingeret 1984). 

The process for those using CASAS 
includes initial identification of basic 
reading and math functional skill levels 
through appraisal (locator) tests designed 
from the CASAS test item bank, develop- 
ment of an instructional plan (including 
reconomendations of commercially pub- 
lished curriculum materials related to 
identified competencies), and assessment 
of specific competencies. 

Critics raise questions about the gap 
between CBAE theory and much of iti 
practice. Although a full consideration of 
the issues they raise caimot be included 
here, programs interested in adopting a 
CBAE approach, as well as those cur- 
rently using CBAE, may find an explora- 
tion of some of these concerns usefiil. 

In practice, some CBAE programs may 
operate from a limited notion of context 
Unlike the NYS External High School 
Diploma program, "context" in otl.sr pro- 
grams can refer to the material in which 
skills are embedded in class, rather than 
the setting, conditions, and purposes a 
particular task might have in real life. 
Collins (1983) notef that assessment in 
those programs is based on paper and 
• pencil multiple-choice tests (and oral 
responses to cues) removed from compe- 
tent action contexti^alized in the real 
world. Therefore, claims for competence 
arising out of CBAE are legitimized 
within its own system yet unsubstantiated 
in the real worid. The assessment, there- 
fore, is in most cases still a test given 
under the peculiar conditions of the class- 
room. The sole reliance in some models 



on paper and pencil (or computer) tests 
removes the key theoretical notion of 
••successful functioning in life roles" 
(Ratcliff 1983, p. 113) from the 
assessment process. 

It has been suggested by some that CBAE 
carries with it the '•false aura of exactness" 
(Collins 1983, p. 175). As with standard- 
ized tests, the appeal of competenqy-based 
assessment is based on its claims for 
objectivity. An adult educator interviewed 
by Elsa Auerbach (1986) explains: 

There's an illusion that by calling 
something competency-based you're 
removing your own prejudices from 
the processes of teaching and 
assessing-and that is a dangerous 
illusion, (p. 421) 

Thus, competency-based assessment car- 
ries with it a set of beliefs and assump- 
tions that are not appropriately regarded 
as neutral. 

Others argue that a definitive assessment 
of competency is not possible without 
examining the processes involved in com- 
petent performance. Programs that rely 
solely on standardized tests requiring 
right/wrong answers are mostly unable to 
do this. Alternatively, some competency 
educators do stress the importance and 
value of ••appraisal skills^* (Monjan and 
G^wsner 1979)~learaers' metacognitive 
awareness of how they accomplish par- 
ticular activities-and argue for including 
self-report on the process of accomplish- 
ment within the competency testing 
framework. Others have suggested a 
validity problem inherent in using too 
small a sample of behavior to establish 
competence (Monjan and Gassner 1979); 
they argue that success (or failure) in a 
single performance may be evidence that 
the skill is within the person's capacity, 
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but not evidence that the person can 
typically perform this skill. 

As with materials-based assessment, critics 
argue that the close links between curricu- 
lum and assessment in competency-based 
assessment systems control and restrict 
teaching <^d learning. In some cases 
competencies do emerge from dialogue 
between instructors and learners "in which 
needs are identified collaboratively as a 
result of accumulated trust and experi- 
ence, as an integral part of learning** 
(Auerbach 1986, p. 422). In other 
instances, they are established before 
instruction begins and tied into commer- 
cially produced teaching materials that 
predetermine instruction. Auerbach and 
Collins further question the ability of 
competency-based assessment systems to 
assess critical thinking skills; since such 
skills are less readily quantifiable. 
Fingeret (1984) adds that "increased self- 
worth, so highly valued by ABE program 
participants, cannot be reduced to a set of 
competencies" (p. 30). 

The distinction here seems to be in 
whether competency approaches are 
viewed as instructional/assessment pro- 
cesses with the potential to meet specific, 
limited learning objectives (ibid.) or as a 
complete system. Systems, critics argue, 
circumscribe the roles of teachers and 
learners in establishing, pursuing, and 
assessing their own curriculum, and they 
are designed in ways that lose sight of 
some of the unique opportunities for con- 
textual learning and assessment possible in 
competency-based education. Speakmg of 
competency-based programs in colleges. 
Elbow (1986) sets some preconditions use- 
ful to apply to literacy education. Fh-st, 
teachers must play a major role in deter- 
mining outcomes and assessment proce- 
dures rather than serve merely as the 
transmitters of predetermined curricula 



and assessments. To capture the com- 
plexity of learning, he recommends that 
outcomes themselves extend beyond spe- 
dBc fimct'onal performance objectives to 
be "broad and deeply intelligent, not neg- 
lecting the larger dimensions of human 
growth nor the special dimensions of intu- 
ition and aeativity" (p. 139). Since 
competency-based assessment often is 
employed to assess the effectiveness of 
teaching. Elbow suggests that the feedback 
loop operate in both directions: that in 
addition to assessing teaching by the 
results of competency-based assessments, 
teaching should cast some light on the 
validity of state i outcomes and 
assessments. 

CBAE educators themselves have called 
for improvement in assessment tools 
(RatclifiP 1983). In doing so educators are 
urged to make clear a distinction between 
competency-based systems (in which com- 
petencies are the starting and ending 
point of curriculum development and 
assessment) and competencies as instruc- 
tional strategies, in which they are one 
among many in the process of enabling 
learners to act for change in their lives 
(Auerbach 1986). 



Participatoiy Assessment 

One way of understanding participatory 
assessment is as a process rather than a 
tool or set of tools, distinguished from 
other assessment approaches by its view of 
literacy and literacy education and by its 
ei..phasis on the active participation of 
both adult learners and program staff. 
Participatory literacy education is "based 
on the belief that learners, their charac- 
teristics, aspirations, backgrounds, and 
needs should be at the center of literacy 
instruction" (Fingeret and Jurmo 1989, 
p. 5). Staff in participatory programs thus 



can be said to rely on the perspective 
brought by adult learners Uiemselves in 
designing instruction, defining program 
identity, and collecting assessment data. 
Participatoiy assessment, then, necessitates 
a coUaborative relationship among learu- 
ers and program staff in determining the 
goals, texts, and contexts of assessment, as 
well as in judging its outcomes. 

The view of literacy and literacy education 
within participatory assessment also 
reflects the centrality of the learner. This 
view assumes that literacy practices differ 
within communities and cultures and con- 
sequently are social, political, and eco- 
nomic in natme. Essential to this per- 
spective is the notion that literacy has 
different meanings to different users 
(Holland 1989) and that literacy practices 
involve a wide range of texts, tasks, and 
contexts (Lytle, Brandt, and Vannozzi 
1988; Lytle, Marmor, and Penner 1986; 
Wolfe 1988). What gets assessed, then, is 
not necessarily the same for each learner, 
but rather is determined by learners' goals 
and literacy needs within contexts they 
define. 

in characterizing assessment procedures as 
"participatory," it is useful to widen the 
definition to include program staff. In 
contrast to systems and tests, instruaors as 
well as learners play a role in participa- 
tory assessment. They may participate in 
the development of tools and strategies, 
formalizing for their programs the "infor- 
mal" assessments previously used in indi- 
vidual classrooms. At the least, they are 
involved in dialogue with learners 'vithin 
the assessment process. 

Learner involvement also takes different 
forms in participatory assessment In 
contrast to the circumstances surrounding 
standardized testing, there is a degree of 
learner choice and control over what gets 



assessed: which pieces of work are 
reviewed, which materials are read and 
responded to. Although some work may 
be done during designated "data- 
gathering activities" (Koen 1986a), 
assessment is understood as deriving in 
part from the work akeady done by learn- 
ers within the program as well as in other 
contexts in which the leamer*s education 
is likely to have had an impact. The 
choice of which pieces to show fcr assess- 
ment or which strategies to demonstrate 
belongs primarily with the learner. 
Learners, therefore, are encouraged to 
develop an understanding of the relation- 
ship between a literacy practice being 
assessed and its use in tiieir lives outside 
the progranL Participatoiy assessment 
further emphasizes learners' perspectives 
on their own learning. As a result, self- 
evaluation and the development by learn- 
ers of metacognitive awareness-the abiL'7 
to describe the ways in which they 
approach particular reading and writing 
tasks and the strategies they apply to 
them-play a central role. 

Finally, in some instances assessment 
procedures are designed collaboratively by 
staff and adult learners; when learners 
play a major role in the development and 
refinement of strategies for teaching and 
assessment, the assessment becomes more 
fully participatoiy (Lytle ct al. 1989V In 
such cases, distinctions between assess- 
ment and instruction become deliberately 
blurred; the process of participatory 
assessment becon:es a co-investigation 
into assumptions about literacy, teaching 
and learning, and the nature of progress 
and change. 

Some of the procedures and emphases of 
participatory assessment are siir>i'ar to 
those in competency-based assessment and 
in infortiial, classroom-based assessments 
done by indiwdual insta'ctors. The range 
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of data-gathering strategies used includes 
paper-pencil surveys, interviews, learner- 
selected demonstrations of reading and 
miting, and collections of learners' work 
in portfolios. They yield both qualitative 
and quantitative data indicating learners' 
str<;ngths and strategies rather than their 
needs and deficits. Assessment becomes 
an integral part of instruction and cur- 
riculum development and is itself a form 
of teaching. 

Discussed here are four assessment pro- 
jects that share features of the participa- 
tory approach. Because this is not a 
generic type of assessment, but rather a 
group of loosely related projects seeking 
to involve learners and staff in generating 
more complex portraits of learner accom- 
plishment, some description of each pro- 
ject is necessary in order to identify 
features that seem to be unique to a par- 
ticipatory approach. 



Adult Literacy Evaluation Project 

One assessment process that embodies 
many of the featiu-es of participatory 
assessment is the Adult Literacy Evalua- 
tion Project (ALEP) in Philadelphia. 
Begun as a collaborative research project 
between the Center for Literacy (CFL), a 
not-for-profit, multisite volunteer Uteracy 
organization, and the Literacy Research 
Center of the University of Pennsyh^ania, 
ALEP has developed alternatives to stand- 
ardized tests and grade-level descriptors in 
assessing growth in adult literacy. 

A number of assumptions about adult 
learners and Vl(eracy assessment, derived 
by the development team £rom their 
experience as litera(^ researchers and 
practitioners, informed the development 
of the ALEP procedures. These beliefs 
reflect the philosophy of partidpat07 



education (Lytle, Brandt, and Vannozzi 
1988): 

o Adults come to programs with particu- 
lar goals or objectives. 

0 Adults bring with the .1 perceptions or 
beliefs about literacy and illiteracy, 
reading and writing, teaching and 
learning, that affect Sieir success in a 
program. 

o Assessment procedures communicate 
notions about literacy and about 
relationships between teachers and 
learners. 

0 Adult learners' expectations about 
learning and their own abilities may be 
strongly influenced by what a literacy 
program chooses to assess and by the 
particular methods used. 

Through ALEP's assessment, learners and 
staff can identify and prioritize individual 
goals (make plans), describe the functions 
and uses of literacy in adult learners' 
everyday lives, determine the extent to 
which they are able to perform a range of 
reading and writing tasks and apply vari- 
ous strategies to them, and understand the 
beliefs about reading, writing, and literacy 
that adults bring to the learning process. 
All four of these dimensions-goals and 
plans, practices, strategies, and beliefs- 
are deemed essential in tiie project for 
accurately determining progress in learn- 
ing over thne^Lytle and Schultz 1990). 

To assess adult learners when they enter 
the CFL program and to document 
change and growth over time in each of 
the four dimensions, staff coordinators 
conduct planning coiOPerences with learn- 
ers at regular intervals. Approximately a 
one- to one-and-one-half hour interview, 
the initial planning conference has four 
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major sections. First, to get a sense of the • 
role of literacy in their everyday lives, 
learners describe the variety of contexts in 
which they engage in literacy practices. 
Then, for a record of reading and writing 
strategies and interests, learners choose 
"real-world* and CFL student-written 
materials to read and discuss, and they 
compose their own writing. Word recog- 
nition is assessed by aslang learners to 
read signs that have been photographed 
in the local community. Information 
about concepts of literacy, reading and 
writing, and teaching and leariiing held by 
learners is elicited throughout the assess- 
ment of strategies and expanded through 
other procedures such as interviews, 
lexts chosen by learners are analyzed for 
increasing difficulty, as are the tasks 
involved in reading these texts. Also 
noted is the degree of engagement with 
and critical response to texts, studied 
through the repertoire of questions indi- 
vidual learners ask themselves when they 
read. Finally, learners assess their own 
priorities by responding to a checklist of 
goals developed from those articulated by 
CFL participants over the years. Learners 
indicate which they can already do, which 
are of particular interest to them, and 
which goals are not relevant or important 
to them. The process of goai setting is 
informed by the preceding assessment of 
beliefs, practices, and strategies. Later 
planmng conferences, which may involve 
tutors and teachers, focus on a portfolio 
of accomplishments and work in progress 
which adults bring to the conference and 
analyze in collaboration with the coor- 
dinator and/or tutor or teacher. 

One advantage of the ALEP assessment 
procedures is that they can translate 
directly to suggestions for tutors and 
teachers in planning the instructional 
program. The range of learners' activities 
and interests brought to light during the 



assessment is established in a dialogue 
between staff and the learners themselves, 
and so instruction can be planned to 
address individual aspirations and needs. 
The assessment itself becomes a first 
learning experience at CFL for entering 
adults, who come away with information 
about what is involved in becoming a 
more effective reader and writer and often 
with a broader view of the ways literacy 
may affect their participation in various 
social networks, in community activities, in 
jobs, and in other areas. By inviting 
learners into a dialogue within the assess- 
ment, programs run a lesser risk of formu- 
lating concepts of success at odds with 
those possessed by the learners them- 
selves. Use of ALEP procedures provides 
an educational context for the collabora- 
tive investigation of adult learning and 
serves as one form of practitioner 
research. (For another example, see 
Wolfe 1988.) 

One limitation of the original ALEP 
relates to tutors' participation in the 
process. The assessments are conducted 
by CFL coordinators who have more 
expertise than volunteer tutors in literacy 
education and who been trained to con- 
duct the assessments. The coordinators 
must translate their results for CFL tutors 
who may hold differing concepts of lit- 
eracy, teaching, and learning, and who 
might benefit from participating in or 
observing an ALEP assessment. 

CFL has recently revised its orientation of 
tutors to provide them with experiences 
designed to help them imderstand and 
interpret the ALEP data. Another limita- 
tion of ALEP is that the information it 
produces is not currently quantifiable in 
ways that funders often require. The 
results may also be frustrating to adults 
who have been schooled into grade-level 
descriptors and who therefore expect 
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more traditional concrete "data." Others 
have noted that the greatest problem of 
such ethnographic assessment procedures 
might be "how to provide standardization 
so that the needs of a large and poten- 
tially mobile population can be provided 
for" (HiU and Parry 1988, p. 62). 



California Adult Learner Progress 
Evaluation Process 

An assessment process that shares some 
of ALEP's features is the Cah'fomia Auult 
Learner Progress Evaluation Process 
(CALPEP), a program of the California 
State Libraries/CaMfornia Literacy 
Campaign (CLC) and the Educational 
Testing Service. CALPEP was initiated in 
order to develop a common approach to 
assessment in the 46 library sites 
(Solorzano and Stecher 1987). 

The CALPEP assessment system consists 
of procedures for initial assessment and 
ongoing monitoring of learner progress, as 
well as semiannual tutor reports marking 
individual progress over a set period of 
time or upon exit from the progiam. Its 
audiences are the tutors and learners 
themselves and, twice a year, program 
administrators. CALPEP is process 
oriented and program centered. The 
forms were developed from information 
collected from tutors, coordinators, and 
learners at selected CLC sites. They are 
designed for use by tutors and learners 
' together and are written with the learner 
as the primary audieuce (for example, 
"How o^en do you read these things out- 
side the tutoring session?"). By noting 
and re-checking learners* goals, intentions, 
and personal assessments of ability, the 
greatest changes seen among adult learn- 
ers in programs-shifts in goals and 
perceptions-are documented alongside 
changes noticed by the tutor and learner 



in learner ability at reading and writing 
for different purposes. 

Hie three interview sections of the 
CALPEP procedures investigate learners* 
literacy practices. Learners describe their 
reading and writing habits and goals. 
They also comment on whether the pro- 
gram has helped them in work or in find- 
ing a job, and (for exit assessment) on 
why they are leaving the program 
(Solorzano and Stecher 1987). The 
lessons/materials pages in the Tiitor/ 
Learner Progress Ix)g indicate an attempt 
to encourage tutors to plan their instruc- 
tion according to the goals and progress of 
individual learners. Other strengths 
include a movement away from reporting 
learners' abilities in terms of grade-level 
equivalents and deficits toward a profile 
of what learners can do. L. addition, 
CALPEP validates writing as part of 
literacy. 

As in ALEP, the CALPEP processes are 
time consuming, and coordinators need 
training to conduct the assessments and 
translate their results for tutors and 
teachers. CALPEFs main limitations, 
however, concern the ways in which the 
theory of reading and writing underlying 
data analysis and reporting contradicts the 
learner-centered, participatory nature of 
the Intendews and checklists. On the 
basis of learners' responses in the inter- 
views, and of the work done in tutoring 
sessions, the tutors then define one of five 
reading and writing levels for each learner 
(for example. Level "S" in reading: can 
read simple sentences). These levels 
move liuearly from "readiness" to "applica- 
tion" and rely on a bottom-up, skills-based 
view of literacy. In seeking to make their 
descriptive data • reportable, then» 
CALPEP has developed a scale that is 
ultimately not very different from those of 
more traditional assessment approaches. 
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The model of writing presented in the 
CALPEP levels also assumes i linear 
progression moving from letters to words, 
to sentences, to paragraphs, although the 
ability of both very young and adult 
learners to write lengthy texts without 
much knowledge of skill i has been well 
documented (for exajiple, Kazemek 
1984). 



ALBSU Progress Proflle 

The ALBSU Progress Profile, developed 
by the Adult Literacy and Basic Skills 
Unit in Great Britain, is one example of 
a student progress evaluation where 
learner and tutor together work out aims, 
set goals, plan the work needed to be 
done, and review progress (Holland 1989). 
The Progress Profile is organized in two 
parts. The first is a form consisting of 
five plaiming questions to which learners 
respond periodically as aims and goals 
change. The second part is a progress 
review form, on which learners list specific 
goals and elements of those goals, graph 
their progress, and describe briefly how 
they have used what they have learned. 

As in CALPEP and ALEP, the prim?^' 
audiences for this assessment are within 
the program: managers, instructors, and 
learners. Although little documentation 
of its impact is currently available, 
instructors and learners may benefit from 
the Progress Profile's emphasis on accom- 
' plishment rather than deficit and its view 
of assessment as a reflective Iraming 
activity done by the learner witli tutor 
support Although no baseline data are 
collected, the Profile provides a clear 
schema for recording self-reported impact 
data useful to program managers in tuoni- 
toring the effectiveness of instruction and 
in making decisions about additional 
resources or staff training needs (Holland 



1989). This process is still in a pilot 
stage; further study is needed to deter- 
mine whether its view of progress as a 
linear movement is viable and the ways in 
which tutors and learners might go about 
determining what coiistitutes progress tor 
a particular literacy ^oal. 



Student Progress Evaluations 

Koen (1986a) outlines a process for orga- 
nizing program-wide "student progress 
evaluations." This process views assess- 
ment as the periodic monitoring of learner 
accomplishment At the center of this 
process pro-am staff perform the follow- 
ing tasks (ibid., p. 12): 

0 Identify possible features of perform- 
ance to be assessed in line with pro- 
gram instructional focus, philosophy, 
and methodologies. 

o Review the list to determine priorities" 
to decide which areas are critical to 
include in student progress assessment, 
considering both the short-term and 
long-term needs of participants. 

0 Determine which assessment topics 
need to be included during the initial 
assessment and which ones can be 
incorporated into future monitoring 
activities. 

o Consider aod choose methods for con- 
ducting the assessment, keeping in 
mind that multiple measures make the 
assessment more credible. 

This process emphasizes the development 
of procedures by programs for their own 
use, use.i both qualitative and quantitative 
methods and analysis, and uses informa- 
tion from the assessments in instructional 
planning. However, it does not involve 
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learners in the articulation and prioritizing 
of assessment topics, and it relies mainly 
on tests for indications of reading 
accompUshment 

The complexity of learner accomplishment 
in adult literacy programs ciJls for a 
further mvestigation into participatory 
approaches. Although there are only a 
few examples in the literature describing 
these approaches, ihey desenre attention 
both for their reconceptualization of 
assessment and for the challenges they 
present in program development and 
evaluation. 

literacy educators seeking to design par- 
ticipatoiy assessment are faced with a 
number of program issues. In designing 
an appropriate assessment, they must 
examine critically the assumptions about 
literacy teaching and learning that inform 
their practice. Work on assessment, 
therefore, requires work on instruction 
and curriculum. Since participatory pro- 
cedures place a great deal of the respon- 
sibility for assessment in the hands of 
teachers and learners, progiiLms need to 
plan staff development work that assists 
teachers and tutors in understanding the 
important role assessment can play in 
their classes, in planning and sharing 
assessment/instructional activities, and in 
documenting changes in learners over 
time. The inclusion of qualitative meth- 
ods in participatoiy assessment presents 
opportunities to consider the role learners 
can play in their own assessment: for 
example, in identifying their goals and 
purposes for literacy, in choosing materials 
to be read during an assessment and writ- 
ings to be reviewed, or in providing self- 
reports of their progress in journals and in 
peer interviev/s. Finally, the kinds of data 
collected raise questions about how to 
interpret and use the data, and how to 



make it reportable to funders and other 
interested outsiders. 



Conclusion 

The four approaches reviewed in this 
chapter suggest several criteria for 
deciding which learner attainments to 
assess. Of prime importance seems to be 
the degree of congruence between par- 
ticular approaches and a program's cur- 
ricula and teachmg practices. There are, 
however, several points of view on this 
issue. Some feel that the ideal drcum- 
stance occurs when a test is different from 
learning activities in the program and thus 
attempts to assess abilities or behaviors in 
an objective or neutral fashion. At the 
other end of the continuum is the view 
that assessment and teaching should be as 
closely related as possible and in some 
instances indistinguishable. This issue is 
complicated by approaches that interpret 
congruence as meaning that both instruc- 
tion and assessment should be standard- 
ized and by the expectations learners 
bring to programs about what i£ and is not 
meaningful assessment. Familiar with tak- 
ing standardized tests from their previous 
schooling, some adults find these tests 
extremely anxiety-producing because they 
are so reminiscent of previous failures. 
Others, however, seek validation for their 
progress on measures they associate with 
school success. A related issue is the 
capability of any single measure to cap- 
ture the repertoire of skills and strategies 
an individual needs to accomplish a vari- 
ety of literacy tasks. One possible solu- 
tion is the development and use of multi- 
ple methods that togetht/ provide a more 
complex and thus more accurate portrait 
of learner accomplishment 

A number of important learner outcomes 
have only begun to be addressed iii 
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assessment. Although various approaches 
assess literacy skills, tasks, and practices, 
ody participatory approaches have the 
potential for assessing literacy as critical 
reflection. Only two of the four 
approaches reviewed appear to take into 
account the notion that assessment is both 
process and product: both competency- 
based and participatory assessments in 
theory provide opportunities for learners 
to explore and exhibit behaviois, beliefs, 
md practices over time. None of the 
approaches explicitly addresses the possi- 
bihty that assessment may be conducted 
with groups of learners rather than just 
with individuals. Because most adult 
educators recognize the importance of the 
social nature of learning, many have orga- 
nized their classes in ways that facilitate 
peer learning. Strategies for assessing 
these abilities and experiences would 
contribute to the available options. Yet 
the approaches that attempt to discover 
more about the effects of participation are 
also the most time-consuming, expensive, 
and difficult to administer and report. 

A final issue that these contrasting 
approaches brings to prominence is that of 
audience. Learner assessment information 
is sought by a range of stakeholders, all of 
whom may have different requirements 
for the kind of information needed. It is 



unlikely that the data collected to inform 
instruction would be equally useful for 
describmg the impact of an entire pro- 
gram. Standardized tests have the clear 
advantage of yielding quantitative data 
that are easUy scored and aggregated. 
Yet because of measurement error and 
limitations on their use for diagnostic 
purposes, they are less satisfactory for 
describing individuals' progress and for 
planning programs of instruction. Tests 
are sometimes usw j quite inappropriately 
for determining a person's entry into 
employment or further education or, 
within programs, for placement in a 
specific level of materials or classes. 
Program stafif, fimders, and policy makers 
need to be sensitive to assessments used 
for gatekeeping and to recognize the obli- 
gation to provide adult learners with a 
variety of means to demonstrate success. 

Muhiple methods seem sensible, especially 
since many adult learners describe their 
literacy goals and accomplishments in 
ways that extend beyond narrow defini- 
tions of reading and writing. Learner 
assessment therefore need to be reconcep- 
tualized to reveal more of what prograrjs 
and learners themselves characterize as 
progress or change. 
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IMPLICATIONS AND FUTURE DIRECTIONS 



Evaluation, Assessment, and 
Educational Polii^ 

How evaluations are conducted and how 
the findings &om evaluations are used 
represent not only technical, but 
ultimately political decisions (Talmage 
1982). Because there is a lack of 
consensus about the purposes of literacy 
education, and because the field is by 
definition pluralistic, there can be no 
single definition of evaluation or assess- 
ment nor one view of what makes the best 
program. Evaluation and assessment are 
therefore political because they involve 
choices about focus, procedure, and cri- 
teria, made by program staff^ learners, 
sponsoring agencies, fimders, and policy 
makers. These choices in turn lead to 
decisions that include, for example, 
whether to fund, to increase services to 
particular groups, or to use particular 
materials, and that affect the allocation 
and reaUocation of resources. Evaluatir 
and assessment are clearly central to 
policy making at all levels of the system. 

As has been shown, there are different 
- definitions of evaluation and assessment in 
adult literacy and a variety of models for 
how they should be done. In spite of ttds 
diversity, there is agreement that program 
evaluation and learner assessment should 
be useful to local program staff and learn- 
ers as well as to funders and national 
policy makers. Faced with high rates of 
attrition and persistent questions such as 
whether those most in need of instruction 



are choosing to enroll in the existing pro- 
grams, adult literacy educators need to 
join with researchers and policy makers in 
the exploration of innovative and forward- 
looking approaches to both assessment 
and evaluation. 

Drawing on a review of the available 
literature related to program evaluation 
and learner assessment in adult literacy, 
and to some extent on the literature of 
program evaluation in education more 
generally, 10 critical features have been 
identified that together provide a frame- 
work tor the evaluation of adult literacy 
programs and the learners who participate 
in them. Tbese features and related key 
concepts are summarized in table 3. Al- 
though adult literacy progranu^ share many 
goals and strategies with other educationsd 
initiatives for adults, adult literacy also 
has distinctive social, politic il, psycho- 
logical, and educational featurcts and thus 
distinctive needs for program evaluation. 
The framework, then, is intended specific- 
ally to enable adult literacy programs, 
sponsoring organizations, and policy 
makers to critique current evaluation 
practices andjo plan new dircictions that 
integrate evaluation and assessment needs 
and functions at the local, s^ate, and 
national level. 
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TABLE 3 



OUTICAL FEATURES OF ADULT LITERACY PROGRAM EVALUATION 





FEATURES 




KEY CONCEPTS 


1. 


External and Internal 


0 
0 
0 


Particination bv information users 
Expertise from several sources 
Building capacity of programs for self- 
evaluation 


2. 


Formative and Summative 


o o o 


Evolving over time 
Goal onented 

Program processes and impact 


3. 


Participatory 


0 
0 
0 


Learners 
Staff 

Evaluation as inauirv 


4. 


Theory, Research, and 
Practice-Based 


0 
0 
0 


Questions from all three 
Making assumptions explicit 
Focused on key asnects cf DroE.i'am 


5, 


Critical Reflection 


0 
0 


Philosophy and goals 
Practitioner research 


6. 


Teaching and Learning 
Processes in the 
Foreground 


0 
0 
0 


Suited to context 
Descriptive 

linking processes to outcomes 


7. 


Range of Learner and 
Program Outcomes 


0 
0 
0 


Multiple indicators 
Carefully selected 

Broader definitinn fif literarv 


8. 


Variety of Methods over 
Time 


0 
0 
0 


Formal and informal 
Formative and summative 
Quantitative and qualitative 


9. 


Integrated with Program 
Functions 


0 
0 


Evaluation is institutionalized 
Resource allocation 


10. 


Systematic and Systemic 
* 


0 
0 
0 


Comparable across programs 
Interactive: top down and bottom up 
Strategies for dissemination 
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A Framework for Fsxtgram Evaluation 
and Learner Assessment in 
Adult Literacy Eciucation 

1. Program evaluation in adult literacy 
education should be conductftd both 
externally an^ infiemally. 

Programs need e^t^m^ and mtemal 
approaches to evaJualioii Both assume 
participation b> program managers, st^f, 
and leam^^rs, although their roles will 
vary. Whether or not st^if initiate an 
evaluation^ they can still participate in its 
design, reflect on its findings, and use 
those findings to make program decisions. 
External evaluators offer a different per- 
spective on the program a!id may bring 
needed expertise in evaluation methods. 
By working in collaboration with pro- 
grams, they can develop evaluations out of 
stakeholders' expressed issues and needs 
("utilization-focused," see Patton 1987), 
making it more likely that the evaluation 
findings will acmally be used. Some have 
described the role of an external evaluator 
as an "organizational anthropologist," who 
seeks to imderstand the program from the 
perspectives of staff, participants, and 
other groups affected by the program 
itself (Stecher and Davis 1987). One 
question that arises from the literature in 
adult literacy program evaluation is the 
degree to which an external evaluator 
needs to be familiar with the literacy field. 
It is likely that those who bring an 
extensive background in the theory and 
research base of adult literacy education 
contribute another layer of expertise. 
They may be better able to organize an 
evaluation sensitive to the issues generally 
acknowlevdged as central for most pro- 
grams, such as those related to teaching 
and learning. 

Internal evaluation also plays a critical 
role. Systematic program self-evaluation 



enables programs to make internal policy 
decisions from a richer base of knowledge 
and from a perspective on their own 
program philosophy and practices. In 
some cases, evaluation may be conducted 
by an individual already employed by or 
associated w^th the program and tlius 
alrca^ familiar with program goals and 
personnel. External evaluators can also 
Stssist ongoing r urogram self-evaluation. 
Even a one-time external evaluation can 
put in place strategies for data collection, 
analysis, and interpretation that contmue 
when the formal processes are completed. 
This is especially important for programs 
adopting qualitative methods for the fiirst 
time. Finally, ^^dth better data collected 
internally, adult literacy educators, policy 
makers, and fimders would be able to use 
a wider range of CTOSS^program informa- 
tion in determining more confident direc- 
tions for the field as a whole. 



2. Program evaluation should be both 
formative and summative* 

Adult literacy programs often provide a 
niunber of interrelated educational and 
support functions for participants. Thus, 
they require a range of strategies for 
formative evaluation-conducted to 
improve program processes~as well as 
summative evaluation-designed to make 
decisions about program effectiveness. As 
explained earlier, evaluation tasks may be 
carried out by an individual or group of 
people as internal or external evaluators. 

Some formative evaluations are compre- 
hensive, attempting to understand major 
program features and their interrelation- 
ships (Fingeret 1985). Formative evalua- 
tions are also usefiil when a program 
begins (Hikes 1988b; Turner and Stockdill 
1987) or when some innovation is being 
implemented. They can detail practition- 
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ers' adaptations of their original plans and 
track changes in priorities over time 
(Patton 1987). Examples would be a 
revised method for tutor training, the 
introduction of new technology, or the 
development of participatory approaches 
to learner assessment External evaluators 
in a lormative evaluation see themselves 
as advising program planners and man- 
agers and identifying areas for improve- 
ment; they may become virtually a mem- 
ber of the staff for a period of time. 

Although summative evaluation of pro- 
gram impact on learners is critical, infor- 
mation about outcomes without informa- 
tion about implementation or program 
processes may leave significant questions 
about literacy programs unanswered 
(ACBE 1989). Describing the key pro- 
gram features and activities-tutoring, 
classes, meetings, staff development, 
recruitment, planning, or counseling 
sessions, and gathering data from different 
sources-is an important part of the eval- 
uation (King, Morris, and FitzGibbon 
1987). 

When programs design self-evaluation 
strategies, tiiey may use the term forma- 
tive to refer to the analysis and interpre- 
tation of data collected routinely over 
time as part of the program's system for 
managing and monitoring information, or 
they may establish procedures specifically 
for collecting information about a new or 
revised practice. Summative evaluation, 
. conducted internally by staff and learners, 
with or without external consultation, may 
be focustd on the impact of the program 
at particular points in time, that is, learner 
progress during or at the end of a speci- 
fied time period or course. Because liter- 
acy programs are typically dynamic and 
ongoing, and because the attendance pat- 
terns of adult learners may have their own 
rhythms and logic, the decisions about 



when and how to conduct sun, ...*ve eval- 
uation need to be considered carefully. 

3. Program evaluation and le&kner 
assessment should involve leari^ers 
and staff in a paitlcipatoiy process. 

In a recent volume on participatory liter- 
acy education, Fingeret and Jurmo (1989) 
connect elfectiveness with learner par- 
ticipation in literacy programs. lu 
participatory programs, power is shared 
among learners and stai^ so that learners 
have considerable responsibility for pro- 
gram decision making and operations. 
Learning is built on the "students' knowl- 
edge, skills and experience [which] are 
valued and respected" (p. 1). 

Although the editors of this volume 
describe the participatory model as an 
alternative to the traditional one, and 
although a large number of programs in 
the United States are traditional, there 
are nonetheless compelling arguments for 
the participation by learners in all pro- 
grams in activides related to program 
evaluation. Although literacy programs 
may differ in the extent to which learners 
collaborate with staff in the program 
design, adults themselves are clearly the 
most important stakeholders. Fingeret's 
(1985) study provides an example of what 
can be learned when an evaluation 
attempts to provide insight into both the 
instructional program's dynamics and the 
perspectives of participants. 

Emphasis on program participation can be 
extended to include staff as well as 
learners. Involvement in the design and 
activities of evaluation by staff of literacy 
programs, whether professional or volun- 
teer or both, can contribute to their own 
professional growth and job satisfaction. 
Raising questions about current practices, 
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determining appropriate strategies for and 
collecting data, meeting to ainalyze and 
interpret the information-all constitute 
forms of what is being called practitioner 
or action research. The notion of inquiry 
as a fimdar xntal part of education at aU 
levels suggests that evaluation, rather than 
being relegated to compliance, can 
strengthen a literacy program as it 
becomes a focus for discussion and shar- 
ing across program managers, staff, and 
participants (Hikes 1988b). 



4. Questions for the design of program 
evaluation should be generated firom 
theoiy, research, evaluation^ and 
program practice. 

The questions tha' ive program evalua- 
tions can come Lorn current theory, 
research, and evaluation as well as from 
the problems and concerns articulated by 
practitioners and other stakeholders. In 
Fingerefs (t985) evaluation of North 
Carolina literacy programs, for example, 
one important concern was the extent to 
which statewide programs reflected state- 
of-the-art knowledge about reading and 
the process of learning to read. Thus, her 
evaluation questioned the extent to which 
theory and research had been integrated 
into program practice. Programs planning 
evaluation processes can consult the 
relevant literature-on adults as learners, 
concepts of literacy, and reading and writ- 
ing, for example-to identify questions that 
have value or meaning in a specific pro- 
gram context. Evaluations of other liter- 
acy programs, as well as handbooks, sur- 
veys, instruments, and policy studies, pro- 
vide additional ideas for questions that 
enable program participants to ^'stand out- 
side" the program and look at what is 
happening. 



Evaluation can also be theory based in 
another sense (King, Morris, and Fitz- 
Gibbon 1987). A theory-based evaluation 
focuses on program implementation and 
asks on what tiieoiy of instruction, learn- 
ing, development, or culture the program 
is based. Part of the process thus involves 
making the implicit theoretical base of the 
program explicit so evaluation can then 
determine what activities are critical to 
enacting such a philosophy and whether 
these activities are indeed being under- 
taken at the program. The evaluation 
depends not on testimony, but rather on 
documentation and collection of evidence. 
Staff in a tutor training program, for 
example, may describe it as based on a 
particular view of literacy and learning. 
An evaluation of tutoring would need to 
include a review of training materials and 
visits to tutoring sessions to examine the 
relationship between theory as intended 
and actual practice (Koen 1985). 

Questions that guide evaluations should 
focus on key aspects of the/ program. 
Johnson (1986), for example, describes 
evaluation designs and criteria geared 
specifically to the special features and 
goals of Ubiaiy-based literacy programs 
tiiat may or may not provide instruction 
for adult learners. The ACBE survey of 
literacy education programs (1986) 
describes a set of distinguishing features 
that could be used to generate a set of 
evaluation questions geared to literacy 
and comi lunity involvement The evalua- 
tion of the Technology for literacy Pro- 
ject (Turner and Stockdill 1987) is based 
on questions about the uses of technology 
in individually oriented programs. Design- 
ing questions that focus on key aspects of 
a program may require making fiuther 
distinctions within these broad categories, 
however. Workplace programs not only 
differ from other types of programs, but 
also from each other (BCEL 1987). The 




questions that emerge from practice in 
specific settings will clearly differ. What 
seems important, though, is drawing on 
multiple sources for questions so that 
evaluating individual literacy programs 
occurs within a broader educational con- ' 
text as represented by theory, rcstskrch, 
and evaluation 



5. Program evaluation should Involve 
critical reflection on program 
philosophy and goals. 

To an extent, literacy programs' goals 
have been shaped by federal and state 
funding legislation. Programs often feel 
that in carrying out the intentions of fund- 
ers, tbey have little flexibility in deter- 
mining their own goals. Any examination 
of goals at the local level by necessity may 
be constrained by this larger context. 

The critical examination of goals at the 
program level nevertheless remains a pri- 
ority, though not a simple matter. Al- 
though there appears to be a consensus 
in the literamre that the process of 
progrm evaluation should begin with a 
clear formulation of program goals (Koen 
1985; Lerche 1985a), this may not be pos- 
sible or desirable. In fact, exploring 
different perceptions of a program's goals 
may need to be part of tibe process. 
Research suggests that concepts of success 
and definitions of literacy among diverse 
stakeholders (administration, teachers, 
learners, funders) may differ (Fingeret 
1985). Part of the evaluation may involve 
trying to identify and understand discrep- 
ancies among different viewpomts in order 
to consider the implications of these dif- 
ferences for decision making. 

A program's involvement in critical reflec- 
tion on its practices entails paying atten- 
tion to aspects of the program that puzzle, 



surprise, or raise questions about 
previously unexamined practices. Making 
certain aspects of practice 'problematic" 
(making the familiar strange) requires 
deferring judgments, observing carefully, 
trying to understand phenomena from the 
perspev^tives of others (particularly 
learners), asking questions of inter- 
pretation before questions of cause and 
effect, and in general questioning routines 
in order to seek new understanding of 
familiar situations. As Fingeret (1985) 
and others have pointed out, it is 
extremely important that all involved in 
adult literacy education maintain the 
ability to quest'on the assumptions and 
beliefs underlying practice. 

Program evaluation has typically meant 
measuring the extent to which a program's 
outcomes or attauiments matched pre- 
determined measurable goals. Scriven 
(1972) has proposed an alternative called 
"goal-free evaluation" in which evaluators 
collect data directly on "program effects 
and effectiveness without being con- 
strained by a narrow focus on stated 
goals" (Patton 1987, p. 36). In addition to 
reflecting critically on program goals, 
then, some or all of a literacy program 
evaluation can be more inductive and 
open to prt gram effects that may not have 
been anticipated. 



6. Program evaluation should give 
prominence to the processes of 
teaching and learning. 

The overall goal of program evaluation is 
to provide a reliabV. description of the 
relationships among program processes, 
participants, contexts, and outcomes. To 
arrive at such an understanding, examina- 
tion of a range of program components 
may be ne». £d. Other than the observa- 
tions made in the North Carolina study 
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(Fingeret 1985), however, there is far too 
little information in existing adult literacy 
program evaluations about the processes 
of teaching and learning. Tracking and 
monitoring students and programs are 
clearly important parts of program evalua- 
tion, but the interaction of different pro- 
gram components or functions and &eir 
impact on adult learners, cannot be under- 
stood without a close look at instruction. 
Although a large percentage of literacy 
education in the United States is carried 
out one-to-one by volunteer tutors, for 
example, there is little information about 
what actually occurs in tutoring sessions or 
about the range of approaches to tutoring 
and the effects of these approaches with 
different learners and tutors. Gasses as 
contexts for literacy instruction are simi- 
larly undocumented. 

As m X and more programs develop and 
test Curricula and instructionia strategies 
suited to their particular contexts, careful 
documentation of these efforts becomes 
critical. Reports analyzing these data 
have the potential to inform policy beyond 
the program level. For example, research 
on recent efforts to develop workplace- 
based curricula suggest that workers' 
interests are often not limited to work- 
related learning experiences but also 
include literacy education related to other 
aspects of their lives (Hikes 1988b). 
Other workplace Uteracy programs reflect 
the skills and abilities needed in particular 
workplaces whereas still others use the 
workplace simply as a setting for a tradi- 
tional program. Evaluation studies would 
contribute to a wider understanding of the 
most effective and responsive ways to pro- 
vide workplace education. Community- 
based or community-oriented programs 
(ACBE 1986; Fingeret 1985) may also 
require distinctive structures for curric- 
ulum development and instruction. 



Qualitative methods are particularly 
appropriate to an examination of the 
processes and practices of instruction. 
Through interviews and observations con- 
ducted at the program, for example, exter- 
nal evaluators and program st£^ and 
learners can explore the notions of teach- 
ing and learning in^Udt in the program, 
participants' own views of teaching and 
learning, and how different views of suc- 
cess are discovered and vicall with. Little 
is known about how views of literacy 
manifest themselves in instruction or 
about the impact of different instructional 
approaches on learners. These descrip- 
tions can be generated by an outside ev^- 
uation team or by staff and learners in- 
house, through structures that make cross- 
visitation, observation, and consultation 
possible. Such approaches yield data that 
contribute to a bener understanding of 
the processes and contexts for learning. 
They contrast raC'^**- dramatically with, for 
example, testing, which fails to capture 
even a part of the complex interactions of 
the people involved in teaching and learn- 
ing (Fingeret 1988). 



7. Evaluations should be designed to 
capture a range of learner and 
program outcomes. 

To date, program evaluation in adult liter- 
acy has been based almost entirely on 
learner impact or outcomes; in most cases, 
these outcomes have been largely deter- 
mined by a limited number of indicators, 
including scores on standardized tests of 
reading and data on retention. Some pro- 
grams have relied on indicators such as 
job acquisition or mobility; vhc literature 
contains cautions about using ^uch indica- 
tors as the sole criteria in pro-am eval- 
uation (Fingeret 1984; Hikes 1988b). In 
Chapter 4 were discussed the range of 
learner outcomes typically assessed in 
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adult literacy programs. It is clear that 
outcomes for learners beyond reading 
gains as measured by tests merit more 
emphasis. The participatory approaches 
to learner assessment include documenta- 
tion of some ot these outcomes; more 
complex and sensitive approaches to the 
assessment of learner outcomes, including 
better tests, are cunently being developed. 
Recent program evaluations in adult liter- 
acy (ACBE 1989; Hikes 1988a; Ttaier 
and Stockdill 1987) have also begun to 
investigate a wider range of learner out- 
comes. In these evaluations, literacy is 
defined more broadly to include attitudes 
as well as performance and oral language 
and technolo]^ as weU as reading and 
writing. The emphasis is on reading and 
writing done in various contexts and for 
different purposes outside the program. 
Other outcomes focus on the impact of 
literacy education on learners' lives: 
encouragement c2 children's intellectual 
and academic achievement, participation 
in community activities, increased com- 
munication in the workplace, increased 
self-esteem, and increased self- 
determination. Although the evaluators 
ard program staff were able to locate 
scjie standardized instruments, other 
methods ano tools had to be developed. 
These included individual and focus group 
interviews, learner activity logs, and case 
studies. Evaluations that include a range 
of learner outcomes have the potential to 
produce rich and complex portraits of a 
program's impact, which would be of great 
' use to programs and would also contribute . 
to the knowledge base on adult literacy. 

Examination of a range of additional pro- 
gram outcomes may be needed to develop 
a more complete understanding of pro- 
gram functioning. The evaluation of the 
Technology for Literacy Project, for 
example (Turner and Stockdill 1987), 
focused on staff and tutor outcomes and 



found that the project's emphasis on train- 
ing teachers, administrators, graduate 
students, and volunteers to incorporate 
technology into their programs created a 
"culture of learning" that benefited all 
aspects of the program, especially direct 
service. Through an examination of pro- 
gram components and theu' interrelation- 
ships, the evaluation uncovered an under- 
lying ethos that may now be intentionally 
enacted by the program, suggesting the 
importance in any program of activities 
for staff and le&mers that promote reflec- 
tion and growth. 



8* Program evaluation and learner 
assessment require a variety of meth- 
ods for collecting data over time. 

Methods for collecting data may be formal 
or informal, quantitative or qualitative; 
they include recordkeeping (demographics, 
intake, contact and instructional hours); 
document analysis (written statements of 
program philosophy, curricula, reconomen- 
dations to tutors, teacher/learner journals, 
notes of meetings); surveys, question- 
naires, and interviews, both structured and 
open ended (with current stzH or students, 
former students, applicants, community 
members or leaders, workplace manag- 
ers); and monitoring in relation to other 
local or national efforts (census data, local 
media). Some methods for gathering data 
about ongomg activities may involve the 
audio- or videotaping of teaching, tutor- 
ing, student meetings, or special events. 
In some evaluations, staff and evaluators 
may set up special meetings of tutors, 
teachers, and/or learners to discuss pro- 
gram practices (focus groups). 

There are a nimiber of ways in which time 
becomes a signiricant factor in choosing 
evaluation methods. In adult literacy 
education, course beginnings and endings 
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are to some extern arbitrary boundaries 
and do not necessarily coincide -with adult 
learning patterns. Therefore, attention 
should be paid to the timing of 
evaluations-wben they seem most sensible 
for the outcomei they seek to measure. 
Both evaluation and assessment are pro- 
cesses in which questions and strategies 
may evolve over time. Collecting data on 
day-to-day program activities means that 
the process of evaluating will be a process 
of discovery, a process of learning what is 
happening (Guba and Lincoln 1981). By 
implication, then, some aspects of pro- 
gram svaluation will be ongoing and 
require periodic self-correction to adapt to 
new information. In other words, an 
"evaluation of the evaluation** may take 
place as the process evolves. The North 
Carolina study (Fingeret 1985), for exam- 
ple, demonstrates the power of a respon- 
sive case study evaluation evolving in 
sttiges. A final issue of partiailar 
importance to conducting an evolving 
evaluation is the amoimt of program tune 
needed to do it. If program staff and 
learners are to play a role, the evaluation 
roust be practical and realistic; teaching, 
learning, and management cannot be 
interrupted. Evaluation cannot simply be 
added; rather, resources may have to be 
reallocated to enable staff to part'cipate 
in evaluation as part of liit^ daily 
activities. 



9» To the extent possible, program 
evaluation and learner assessment 
should be integrated nith program 
ftmctions* 

Rather than regard evaluation as a com- 
pliance activity mandated from the outside 
by funders or policy makers, programs can 
institutionalize processes that involve 
learners, staffs and managers in ongoing 
activities of collecting, analyzing, and 



interpreting infomiatioa These self-study 
processes are ccagruent with notions of 
learner participarion in program design 
and management as well as with current 
views of teacher and staff development as 
forms of practitioner research (Berthoff 
1987; Cochran-Smith and Lytle 1990; 
Goswami and StiUman 1987; Johnston 
1987; Kyle and Hovda 1987a,b; LyUe and 
Cochran-Smith 1989; Rudduck and 
Hopkins 1985). Program self-evaluation 
thus becomes an important dimension of 
staff development. 

Integration with program functioning has 
another implication that needs further 
exploration in the field. Although the 
allocation of additional funding for 
program evaluation is needed, the re- 
allocation of resources inside programs to 
include the ongoing collection and discus- 
sion of data is at least equally important. 
Program resources include the time of 
managers, staff, and adult learners as well 
as use of material resources such as com- 
puters and recording devices. This sug- 
gests that rather than conceive of evalua- 
tion as an option to be added at a particu- 
lar point in time, literacy educators should 
be^ to reconstruct all aspects of their 
programs to include it. According to 
organizational theory, the structure of 
such programs is then set up for "learning 
to learn" (Morgan 1986). The plan for 
incorporating evaluation more fully into 
practice needs to be realistic, however, in 
terms of resources and constraints. 



10. Program evaluation should be sys- 
tematic and systemic, enabUng 
stakeholders to make comparisons 
within and across programs and 
contexts. 

The processes and products of program 
evaluation need to serve a variety of 
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purposes and meet the needs of diverse 
audiences, both inside and outside the 
program. When internal program evalua- 
tion processes involve the systematic 
collection of reliable data, these results 
can be more readily used and compared 
across programs, as in the Massachusetts 
Workplace Education Project design cur- 
rently being implemented (Hikes 1988a). 
Large databases (such as AIJES in New 
York Qty) have the potential to make 
useful dly- and statewide program com- 
parisons as well. Bejrond the city and 
state level, national research centers, 
umbrella organizations, and Washington- 
based policy makers can play a role in 
coordinating program research and eval- 
uation initiatives. The quality of the data 
that are collected and compared, however, 
is determined primarily at the local level, 
so that emphasis on program-centered 
evaluation strategies remains the foremost 
requirement. The integration of evalua- 
tion with programmatic functions would 
create the conditions for reflection on 
practice, critique, and informed change. 
There is little evidence that without 
knowledgeable participation at the pro- 
gram level, top-down monitoring can have 
a significant impact on the quality of 
educational programs. 

When systems are put in place for local 
program evaluation (as in Hikes 1988a), 
stakeholders inside and outside the 
program can begin to communicate. 
More interaction among different levels of 
deliveiy systems can lead to policy 
informed by the realities of program 
practice. With richer data collected more 
systematically, funders and policy makers 
will have better information for dissem'- 
nation and decision making. Interacdvt 
rather than top-down evaluation systems 
recognize that policy is actually made at 
all levels, that everyone (including 
teachers, tutors, program managers, and 



adult learners) needs more data on which 
to make decisions. Some different types 
of data will be needed at different levels, 
and although the designs for program 
evaluation will differ from site to site, 
there will still be a greater likelihood of 
comparing efturts. 



Conclusion 

As a synthesis of the current literature, 
this framework suggests directions for the 
field that have the potential to address the 
three purposes of evaluation outlined in 
Chapter 1. 

The first purpose is accountability, which 
some studies suggest should be a 
"mutual" accoimtability in which program 
sponsors and funders work collaboratively 
with program staff and learners to design 
and implement s^fstems for evaluation that 
are interactive and that generate meaning- 
ful data appropriate to all levels of the 
system. Furthermore, recent program 
evaluations (ACBE 1989) present models 
for agencies and umbrella organizations to 
offer evaluation as a service to programs 
rather than simply requesting information 
from them. 

The second purpose for evaluation relates 
to the improvement of practice. When 
programs engage in activities related to 
evaluation and assessment, there are 
important consequences for their daily 
work. Fingcret (1984) calls for adult 
literacy educators to investigate their 
efforts more thoroughly and on an 
ongoing basis. Evaluation and assessment 
are forms of inquiry and staff develop- 
ment. They are similar to action or prac- 
titioner research. In the field of adult 
literacy, learners can take a prominent 
role in discussions about program design 
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and implementation and about staff 
development. 

'iTie third purpose for evaluation the 
generation of new knowledge about adult 
literacy educatioa Program-based evalua- 
tion studies and learner assessment can 
promote dialogue among practitioners 
within and across programs and can be 
furthei enhanced by relationships with 
consultants and evaluators from universi- 
ties. Networks formed around issues and 
practices of evaluation are in a position to 



play a critical role in setting policy. In 
this way practitioners and learners will 
have the data to .support field-based posi- 
tions on the purposes and impact of liter- 
acy education, and university researchers 
will have opportunities to work collabora- 
tively with adult liters 7 educators and 
learners to address critical problems in 
tiie field. When new knowledge is gener- 
ated through these relationships, effective 
dissemination of program evaluations and 
learner assessment becomes a priority. 
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